RE: iSCSI: RE: Framing Discussion

[Wayland Jeong <wayland@troikanetworks.com>]

>> Yeesh!! Thank goodness all I have to do is drop-in an embedded processor
>> into our chip. I'll let the firmware folks deal with this problem.
>> Certainly, this path does not have to be high-performance since
>> we are going into congestion control anyway, but we have to deal with it.
>> We can keep a
>> context stack for the current open TCP sessions which contain mappings of
>> TCP sequence numbers to specific CDB context locations (either command or
>> offset within the gather list). We can keep this stack as deep as the
>> maximum number of outstanding (i.e. not ack'd) TCP segments which for
>> Randy's example (1.25Gbs and RTT of 100ms) is not too bad (about 8K
>> entries). We can recover the contexts needed to re-build this missing TCP
>> segment and re-construct entire PDU(s) so that any necessary
>> digests can be re-calculated. We can then stage this data in memory
>> somewhere and pull-out the exact TCP block that we need to re-send.
>> Lovely.
>
> You forget the most important fact of an iSCSI adapter.  The microcode
does
> the transmit knows exactly how it creates a TCP segment.  First, it would
> not mix multiple PDU's into a single segment.  
>
Yes. The iSCSI adapter can make this choice. But, my concern is in talking
to a proxy like a firewall that terminates TCP. Such a proxy may and
probably will re-package your TCP stream especially if you are sending less
than MSS size segments which you inevitably will if you are trying to align
PDU's at the sender. If a proxy re-bundles your TCP stream and that stream
hits the target with a missing segment, that segment could cross alignment
boundaries. Normally, this will not be the case, but if you design an iSCSI
TOE that cannot handle TCP ACK's with non-PDU aligned sequence numbers then
your TOE has some inherent limitations. At a minimum, it shouldn't break.
Thus, some amount of context and an ability to handle the situation, albeit
on a very slow path, is required.

> Second, there is no TCP stack
> in the iSCSI adapter.   While a command or status PDU is created on the
fly,
> all data PDUs are retransmitted from the application buffers which are not
> freed until an exchange is complete. 
>
I'm not sure what you mean by application buffer. We want to immediately
free the NIC buffers since one of the goals discussed in Randy's framing
document is to avoid having a BWDP worth of high-speed memory on the NIC.
Current Fibre Channel HBA's are single chip solutions with no off-board
memory and only a handful (4-8) transmit FC frame buffers on-chip. If you
are referring to the buffer cache, then you have the same problem which is
how to map the TCP sequence numbers back to your SCSI protocol-level
constructs.

> Third, there is a separate TCP stream
> to each connection.  The task of retransmit is to find the exchange number
> by using the end-point and segment sequence number. The microcode keeps
> track the starting sequence numbers for multiple open exchanges.  However,
> one can take an easy way out by having only one open exchange per
endpoint.
> Of course, this slows things down.  But, don't be too surprised that some
> implementation might just do that.   As I said before, to keep track of a
> few thousand exchanges, there is a lot of memory needed.  However, the
> algorithm to go through the exchange context to retransmit a missing
segment
> is straight forward and does not take a rocket scientist!  (Or should I
say
> an iSCSI scientist?)
>
Yes, it does simplify things if you only have a single outstanding I/O, but
speaking from experience, your performance will stink.

> 
> In summary, transmit is easy because you got to call the shots as long as
> rules are followed.  Can't say the same for receive when old TCP
> implementations must be honored.
>
Thanks again for the input. This discussion really isn't too relevant to the
efforts by this group to standardize a mapping of iSCSI to TCP. These
implementation details are more about the complexity associated with
designing high-performance implementations of iSCSI. By high-performance, I
mean implementations that can meet or exceed current Fibre Channel
performance and can be brought to market at cost parity with FC HBA's.

-Wayland