RE: iSCSI: No Framing

["Peglar, Robert" <robert_peglar@xiotech.com>]

De-lurking...

Right now, I'd say disk-disk backup/restore and mirroring.  The individual
disk devices of today can perform at ~70 MB/s.  One need only gang 14 of
these together, streaming, to reach nearly 1000 MB/s, or 1 GB/s, or 8 Gb/s.
The disk devices of tomorrow are going to perform at rates approaching 100
MB/s per spindle.  Only ten devices are then necessary.

This would be a single TCP stream (one initiator, one target, many LUs).

If one has to mirror (/move) 10 TB in a reasonable amount of time, say, then
a stream of 1 GB/s is not only necessary, but paramount.  Even at that rate,
it would still take nearly three hours to move just 10 TB.  In a
business-critical recovery situation, three hours is an eon.

Don't think application (CPU/memory in a server) to storage.  Think storage
to storage.

Rob

Rob Peglar
Corporate Architect
XIOtech Corporation, a Seagate Company
(314) 308-6983


>-----Original Message-----
>From: Prasenjit Sarkar [mailto:psarkar@almaden.ibm.com]
>Sent: Sunday, February 03, 2002 9:23 PM
>To: THALER,PAT (A-Roseville,ex1)
>Cc: ips@ece.cmu.edu; WENDT,JIM (HP-Roseville,ex1); 
>owner-ips@ece.cmu.edu
>Subject: RE: iSCSI: No Framing
>
>
>
>Hi Pat,
>
>I had a few questions about your 10 Gbps example. If we assume 
>that this is
>a single TCP stream, I would like to know which application in 
>the near and
>forseeable future can sustain a 10 Gbps data rate for a 
>reasonable amount
>of time? Or if they are multiple TCP streams, are you assuming 
>packet drops
>in all streams simultaneously?
>
>Thanks,
>
>
>   Prasenjit Sarkar
>   Research Staff Member
>   IBM Almaden Research
>   San Jose
>
>
>
>                                                               
>                                            
>                      "THALER,PAT                              
>                                            
>                      (A-Roseville,ex1)        To:       
>"WENDT,JIM (HP-Roseville,ex1)"                    
>                      "                         
><jim_wendt@hp.com>, ips@ece.cmu.edu                        
>                      <pat_thaler@agile        cc:             
>                                            
>                      nt.com>                  Subject:  RE: 
>iSCSI: No Framing                             
>                      Sent by:                                 
>                                            
>                      owner-ips@ece.cmu                        
>                                            
>                      .edu                                     
>                                            
>                                                               
>                                            
>                                                               
>                                            
>                      02/03/2002 03:01                         
>                                            
>                      PM                                       
>                                            
>                                                               
>                                            
>                                                               
>                                            
>
>
>
>Jim,
>
>In answer to your questions:
>
>Agilent is planning to implement framing. We view both FIM and
>COWS as having about the same utility so we would implement
>whichever one went into the standard.
>
>A smoothing buffer on the chip is feasible wrt your point 2.
>There are some configurations that would use external memory.
>
>Also, one concern is that the very situation where one would
>need large window size for efficiency - high bandwidth long
>distance communication - is where out of order receipt becomes
>increasingly likely so the amount of memory for this situation
>could push up cost to an excessive extent.
>
>Reducing the amount of traffic that has to be shunted to an
>external memory affects the bandwidth that needs to be provided
>for that memory. If we can handle most PDUs internally then the
>external memory doesn't have to be as wide and as fast. For
>instance, if a drop means that the iSCSI PDU with the drop
>is put into external memory then that takes less memory bandwidth
>than if a drop means the whole data stream for that session will
>be going into the buffer until the hole is filled.
>
>This decision also effects latency after a drop and required
>backplane bandwidth.
>
>Let's say one is on a 10 Gbit/s extreme long distance WAN
>connection and that there is a drop. If the round trip delay
>for getting the whole filled is 100 ms, then without framing,
>one could have about 1 Gbit stored in the local buffer memory
>by the time the hole is filled. One will have to process all
>of this through iSCSI and across the backplane before one can
>deal with the new traffic coming in. If the backplane data rate
>is closely sized to the external data rate, an extra 100 ms latency
>has just been inserted into the path until traffic slacks off.
>(Congestion control might mitigate this to the extent that one
>is talking to just one source, but with multiple sessions, one
>will still have to keep up.) To get the buffer emptied to make
>space for further drops and to get the latency back down, one
>will have to size the backplane bandwidth wider and have the
>iSCSI engine process at a higher data rate.
>
>With FIM or COWS, one can end the incident with much less data
>in the buffer as one processes PDUs after the hole while waiting
>for it to be filled. Then, when the hole is filled one just has
>to process a small amount of data to catch up.
>
>Regards,
>Pat Thaler
>
>-----Original Message-----
>From: WENDT,JIM (HP-Roseville,ex1) [mailto:jim_wendt@hp.com]
>Sent: Tuesday, January 29, 2002 10:47 PM
>To: ips@ece.cmu.edu
>Subject: iSCSI: No Framing
>
>
>
>So, it would be good to hear from several iSCSI
>NIC/chip implementors who:
>- have or plan to implement FIM or COWS (or some
>other framing mechanism) and take advantage of it to
>minimize demands on on-NIC buffer memory
>bandwidth/quantity.
>- believe that all-buffers-on-chip solutions are
>feasible and valid (wrt the points above, including
>#2)
>- can quantify the memory/pin/power/space cost
>savings for all-buffers-on-chip solutions
>
>Jim
>
>
>
>