RE: iSCSI: No Framing

[Dennis Young <dyoung@rhapsodynetworks.com>]

Hi Rod,
Your are correct for the situation where there is only one iSCSI session.
One of the benefits of framing is that when there are multiple sessions, 
a hole in one session won't hold up the other sessions.
Dennis

>-----Original Message-----
>From: Rod Harrison [mailto:rod.harrison@windriver.com]
>Sent: Monday, February 04, 2002 8:36 AM
>To: THALER,PAT (A-Roseville,ex1); WENDT,JIM (HP-Roseville,ex1);
>ips@ece.cmu.edu
>Subject: RE: iSCSI: No Framing
>
>
>
>	Framing of any kind is not a panacea allowing for 
>almost buffering
>free target implementations, it can only move the buffering from the
>transport layer up to the iSCSI layer. Remember that the target MUST
>guarantee commands are delivered to SCSI in the command sequence
>number order. This means if a dropped packet contains an iSCSI command
>PDU you are going to have to buffer until the CmdSN hole is plugged.
>
>	Consider the example of an initiator filling the command window,
>perhaps sending CmdSNs 1 through 100, possibly along with immediate or
>unsolicited data for some of the commands. If the packet dropped
>contained the command PDU for CmdSN 50 then the target iSCSI layer
>must buffer CmdSNs 51 through 100 and their associated DATA_OUT PDUs.
>Since you can continue to process any DATA_OUT PDUs for CmdSNs 1
>through 49 the buffering required is reduced, but not eliminated,
>
>	Only if the packet dropped contains DATA_OUTs and no 
>command PDUs can
>you continue processing past the hole while waiting for it to be
>filled. You can R2T for data associated with commands after the hole
>but that serves only to potentially decrease recovery time, not reduce
>buffer requirements.
>
>	Don't get me wrong, I'm not saying framing isn't a good 
>thing but it
>is only one piece of the puzzle.
>
>	- Rod
>
>-----Original Message-----
>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
>THALER,PAT (A-Roseville,ex1)
>Sent: Sunday, February 03, 2002 11:01 PM
>To: WENDT,JIM (HP-Roseville,ex1); ips@ece.cmu.edu
>Subject: RE: iSCSI: No Framing
>
>
>Jim,
>
>In answer to your questions:
>
>Agilent is planning to implement framing. We view both FIM and
>COWS as having about the same utility so we would implement
>whichever one went into the standard.
>
>A smoothing buffer on the chip is feasible wrt your point 2.
>There are some configurations that would use external memory.
>
>Also, one concern is that the very situation where one would
>need large window size for efficiency - high bandwidth long
>distance communication - is where out of order receipt becomes
>increasingly likely so the amount of memory for this situation
>could push up cost to an excessive extent.
>
>Reducing the amount of traffic that has to be shunted to an
>external memory affects the bandwidth that needs to be provided
>for that memory. If we can handle most PDUs internally then the
>external memory doesn't have to be as wide and as fast. For
>instance, if a drop means that the iSCSI PDU with the drop
>is put into external memory then that takes less memory bandwidth
>than if a drop means the whole data stream for that session will
>be going into the buffer until the hole is filled.
>
>This decision also effects latency after a drop and required
>backplane bandwidth.
>
>Let's say one is on a 10 Gbit/s extreme long distance WAN
>connection and that there is a drop. If the round trip delay
>for getting the whole filled is 100 ms, then without framing,
>one could have about 1 Gbit stored in the local buffer memory
>by the time the hole is filled. One will have to process all
>of this through iSCSI and across the backplane before one can
>deal with the new traffic coming in. If the backplane data rate
>is closely sized to the external data rate, an extra 100 ms latency
>has just been inserted into the path until traffic slacks off.
>(Congestion control might mitigate this to the extent that one
>is talking to just one source, but with multiple sessions, one
>will still have to keep up.) To get the buffer emptied to make
>space for further drops and to get the latency back down, one
>will have to size the backplane bandwidth wider and have the
>iSCSI engine process at a higher data rate.
>
>With FIM or COWS, one can end the incident with much less data
>in the buffer as one processes PDUs after the hole while waiting
>for it to be filled. Then, when the hole is filled one just has
>to process a small amount of data to catch up.
>
>Regards,
>Pat Thaler
>
>-----Original Message-----
>From: WENDT,JIM (HP-Roseville,ex1) [mailto:jim_wendt@hp.com]
>Sent: Tuesday, January 29, 2002 10:47 PM
>To: ips@ece.cmu.edu
>Subject: iSCSI: No Framing
>
>
>
>So, it would be good to hear from several iSCSI
>NIC/chip implementors who:
>- have or plan to implement FIM or COWS (or some
>other framing mechanism) and take advantage of it to
>minimize demands on on-NIC buffer memory
>bandwidth/quantity.
>- believe that all-buffers-on-chip solutions are
>feasible and valid (wrt the points above, including
>#2)
>- can quantify the memory/pin/power/space cost
>savings for all-buffers-on-chip solutions
>
>Jim
>