Re: TCP Framing (considered helpful?)

["Mallikarjun C." <cbm@rose.hp.com>]

John,

Thanks for reiterating your concerns.

You make a case that non-marker software implementations can 
push marker-enabled hardware HBAs into becoming more expensive.
May be, I would have thought that a non-marker hardware sender
is more likely to push a marker-enabled hardware receiver to 
be expensive...

Anyway, isn't this ultimately a performance-cost tradeoff though?
That was the point of my message, in particular pointing out that
there are no interoperability concerns.
--
Mallikarjun 


Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668	Hewlett-Packard, Roseville.
cbm@rose.hp.com



>Mallikarjun,
>We are massively mis-communicating.
>
>The only issue with Markers/Framing is when SW clients use Markers/Framing.
>The first issue is usually performance on the SW side.  I do not think that
>Markers are a big performance issue.  However, I do not have a good handle
>with Framing's overhead.  However, the other major consideration is --
>when will the functions be supported such that we can exploit them.   The
>Markers are a function of iSCSI, and the Framing (Warp) is a function of
>TCP/IP, and hence a concern about the implementation window for relying on
>the Framing availability.
>
>Markers and Framing were never an issue with HW, we could always implement
>them when we create the HBAs.  However, for the function to be useful in
>reducing the implementation cost of HBAs, the Clients must have functions
>that match.  Many of these clients will be SW implementation of iSCSI on
>Desktops and Laptops.  Therefore, the HW vendors will need to insure that
>there is universal support for either Framing or Markers so that the
>implementation on the HBA can exploit that capability.  If it turns out
>that they can not depend on rapid implementation of the function on all the
>SW Clients, then the HW adapter must assume that the function is not
>universal and therefore, they can not depend on it, and must build a more
>expensive HW HBA.
>
>We can cause the Marker to be Mandated as Must Implemented, so we can
>assume universal availability of Markers.  We can not mandate the
>implementation of Framing in all TCP/IP environments.  Therefore, the HW
>HBAs can not assume that they can use that feature.  Hence, they must build
>the more expensive HBA.
>
>Yes, I support Framing and Warp, but the above shows that it can not be
>used to permit the building of a lower cost HBA (for some long time).  On
>the other hand if we can mandate Markers, then the lower cost HW HBA can be
>built, and then they can still negotiate higher performance by negotiating
>for Framing/Warp, and many times they will find a compatible partner and
>then they will function at their optimal rate.
>
>Perhaps we are on the same page now??
>
>.
>.
>.
>John L. Hufferd
>Senior Technical Staff Member (STSM)
>IBM/SSG San Jose Ca
>(408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
>Internet address: hufferd@us.ibm.com
>
>
>"Mallikarjun C." <cbm@rose.hp.com>@ece.cmu.edu on 05/23/2001 02:54:39 PM
>
>Please respond to cbm@rose.hp.com
>
>Sent by:  owner-ips@ece.cmu.edu
>
>
>To:   ips@ece.cmu.edu
>cc:
>Subject:  Re: TCP Framing (considered helpful?)
>
>
>
>John,
>
>Two comments.  I may have misunderstood what you're saying since
>your initial comments imply framing=warp/RDMA/messageboundary,
>and towards the end, you seem to imply framing=markers.
>
>- It's true that implementing iSCSI-level markers does not require
>  modifying TCP/IP stacks.  But using the markers (I mean to use it
>  to effectively steer) requires TCP/IP changes!  It is a separate
>  matter that it is offloaded onto the NIC.
>
>- You seem to implicitly suggest that WARP (when it becomes available)
>  must go into host software stacks to be useful, and hence cannot be
>  done in the right timeframe for iSCSI.  Since one could envision
>  offloading WARP onto the NIC as well, I assume you're hinting
>     a) either at interoperability issues between software
>           and hardware iSCSI implementations relying on WARP,
>        b) or at interoperability issues between hardware and
>           hardware implementations.
>
>  iSCSI currently mandates a "no sync and steering" mode which
>  ensures interoperability in either case.  Perhaps you were really
>  concerned about performance in case (b) then?
>--
>Mallikarjun
>
>
>Mallikarjun Chadalapaka
>Networked Storage Architecture
>Network Storage Solutions Organization
>MS 5668   Hewlett-Packard, Roseville.
>cbm@rose.hp.com
>
>
>>Replies in text below (between [Huff] and  [/Huff]  ).
>>
>>.
>>.
>>.
>>John L. Hufferd
>>Senior Technical Staff Member (STSM)
>>IBM/SSG San Jose Ca
>>(408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
>>Internet address: hufferd@us.ibm.com
>>
>>
>>Stephen Bailey <steph@cs.uchicago.edu>@ece.cmu.edu on 05/21/2001 06:50:41
>>AM
>>
>>Sent by:  owner-ips@ece.cmu.edu
>>
>>
>>To:   ips@ece.cmu.edu
>>cc:
>>Subject:  Re: TCP Framing (considered helpful?)
>>
>>
>>
>>John,
>>
>>> I think we must depend on Markers to insure that everything can operate
>>at
>>> top speed, and at the lowest cost.
>>
>>A key question is whether markers actually ensure that everything
>>operates at `top speed, and at the lowest cost'.
>>
>>Matt thinks so.  I (and, presumably those who wrote the framing
>>document) think not.
>>
>>[Huff] I do not think you can say that.  I also support framing (Warp), as
>>a much more elegant solution, but I find it inappropriate to depend on it
>>actually happening, and being made available in the various OSs in the
>time
>>frame we need.  I do believe, that over time it will be made available,
>and
>>is a better approach for all TCP/IP applications that can use it. [/Huff]
>>
>>My issue is not even with `lowest cost'.  I don't believe markers will
>>allow you to run at top speed.  Specifically:
>>  1) I doubt the feasibility of implementing the control required for
>>     an eddy buffer (where you store data you can't place) at 10G.
>>     Admittedly, the validity of this claim can't really be assessed
>>     without actually working the implementation, so for 99% of the
>>     list participants (myself included) this is a `yes it is, no it
>>     isn't' point.
>>
>>   [Huff] I believe this has had much more work done on it then you
>>      think.  I have personally stepped through the proposals from
>>      several vendors that are working on this option for their HW HBAs.
>>      Usually, because of the iSCSI PDU headers, the data/commands
>>      can be placed directly into the SCSI Host buffers, almost every
>>      time. Only when the PDU headers arrive slightly out of order
>>      (do to normal routing) are the packets unable to be placed
>>      directly into the Host buffer.  And that requires some, but only
>>      a small amount, of buffering space.
>>      It is the packet drops that occur on PDU headers, and resultant
>>      error retries, that cause the need for large amounts of "on
>>      HBA/chip" buffering.
>>      So by using Markers, these HW iSCSI HBAs can limit the amount of
>>      buffering on the chip/HBAs. [/Huff]
>>
>>  2) an eddy buffer solution requires some substantial speed-up in
>>     both the NIC data path, and MOST IMPORTANTLY: the host bus.  In
>>     order to unload the eddy buffer while still handling incoming
>>     traffic at line rate, clearly the host bus bandwidth must be >
>>     line rate.
>>
>>     [Huff] This is not an effect of an eddy buffer solution, it is a
>>     fact that every TCP/IP NIC has to deal with.  Especially at the
>>     new Speeds.  Our current PCI buss will not support 10 Gigabit,
>further
>>     PCI-X will not support it either, even PCI-DDR does not fully support
>>     the full data rate.  So it needs to rely on the TCP/IP window
>>     management.  The only other thing you can do is drop the packets.
>>     this clearly makes the problem worse. [/Huff]
>>
>>
>>I know of at least one general purpose framed solution operating at
>>10G which has been available for >3 years (SGI's GSN/ST/XIO NIC).  I'm
>>sure there are others.
>>
>>I can't imagine there's any argument that a framed solution would be
>>voted `most likely to run fast and be cheap'.  Every storage network
>>and cluster interconnect has been designed that way since antiquity.
>>
>>The key tradeoff involves the OS vendors, and I'm wondering why we're
>>speaking for them.  The question IS, how much more work is it to
>>introduce TCP framing over and above what is required to insert iSCSI
>>into their network framework.  My experience from writing NIC and
>>storage drivers for many commercial UNIX-family OSes is:
>>  1) it's an easy and well defined process to insert a new SCSI
>>     transport driver into the SCSI stack.
>>  2) it's hard and poorly defined process to insert ANYTHING into the
>>     network stack.
>>[Huff] I think you are making my point.  This is the problem with SW
>>Stacks.  That is why I believe that it will take a very long time for
>>the various vendors to include such changes into their "bet you business"
>>TCP/IP SW Stacks.  The point that Matt and I have been trying to make
>>is that most OS vendors are NOT creating the iSCSI HW HBAs (NICs).
>>These iSCSI HW HBAs (NICs) have the TCP/IP completely on the HBA, and
>>they have added the iSCSI processing also so that they can steer the
>>packets directly into the approprate SCSI Host buffers. Adding either
>>Markers or Framing into the iSCSI HW HBAs is not a big problem.  It is
>>only a problem of getting Framing (timely) into Host TCP/IP Stacks.
>>[/Huff]
>>
>>Networking has historically been a user-mode activity.  Architected
>>services are only provided to user mode programs.  Kernel clients have
>>been few and far between and so are handled on a case-by-case basis.
>>For example NFS.  Every OS has hacks to make NFS run fast, but they
>>are not stable interfaces for general purpose use.
>>
>>Even Solaris' SysV-derived STREAMS stack, which is intended precisely
>>to provide flexible, crisp interfaces for kernel network clients, does
>>not document the relevant (IP stack) intermodule interfaces.
>>
>>I know that there are more and more kernel network clients, but they
>>are coming either on fluid platforms (e.g. linux), in which case the
>>argument of `it'll take too long to get OS support' doesn't apply, or
>>they are vendor-supplied, in which case a performance iSCSI solution
>>in ANY form may take a while, and the choice of framing or markers
>>isn't going to make a difference.
>>
>>[Huff] I think you are saying something I agree with and something I
>>do not agree with.  That is, that software changes to TCP/IP in the
>>various "Bet you Business" OSs, will take some time.  However, it is
>>not true that new iSCSI device drivers will take very long.  Two types
>>are being created today.  By Cisco, IBM, Intel, etc. These types are
>>iSCSI DD that make calls to normal TCP/IP stacks, and the DD that
>>are being written by the iSCSI HW HBA vendors.  These do not require
>>the OS vendor to do anything special.  This is happening NOW,
>>(Check with CISCO, Intel, and IBM (me?)).  The last thing we want
>>is to depend on a TCP/IP change to get in the
>>way of our momentum. [/Huff]
>>
>>I can't say squat about the architecture of Winsock, but the fact that
>>there is a Microsoft author of the framing proposal who seems very
>>serious about supporting framing and RDMA as quickly as possible
>>suggests that framing support should be available on Windows very
>>soon.
>>
>>[Huff] My following statements are not meant as a negative of Microsoft.
>>However, they and all producers of Key complicated new Software do
>>not quickly bring these to the general market in a way that is as
>>pleasing to HW vendors as HW vendors would like.
>>
>>I believe that Microsoft's heart is in the right place on this issue,
>>and that they will do the right thing with framing, over time.
>>But it is not clear in what release that will be shipped, nor what support
>>pack it will be included.  Also it is not clear how the support
>>will be handled for current Win2k, WinNT etc.
>>
>>This is why I think we should have Framing a Must implement
>>and an Optional to use.  It is the easiest thing for SW to
>>create, and brings the needed cost reduction to iSCSI HW and
>>it is completely under our (iSCSI protocol) control.
>>[/Huff]
>>
>>
>>
>>Steph
>>
>>
>>
>>
>
>
>
>
>
>