Re: Framing Discussion

[Mohan Parthasarathy <Mohan.Parthasarathy@eng.sun.com>]

> >
> >Aligned to what?  With iSCSI there is a PDU header that must be stripped.  I
> >assume you expect this header to vanish?
> 
> Headers often "vanish" via header / data split - been there, done that many 
> years ago.  Many others do it today for variety of workloads operating over 
> an IP-based protocol suite including TCP (e.g. NFS).
> 
How do  you split the header if you don't know where the header starts ?
You could have packet drops. Moreover the iscsi header can start
anywhere in the stream of data.

If every TCP segment has a iscsi header, which is a waste, then this
problem is relatively simple in identifying which iscsi command 
a given TCP segment belongs to and also indexing into the right
offset inside the buffer.

-mohan
> > > >interface.  The desire is to keep the buffers on the NIC small and thus
> > > >allow out of sequence processing of the TCP stream.  This must
> > > be seen as a
> > > >modification to the normal TCP implementation.  Such operation should
> > > >include a complete description of the API to allow consideration of NIC
> > > >design, inter-operability and security requirements.
> > >
> > > Need to separate out completion and ACK generation semantics from how and
> > > the order that data is DMA'ed into the target buffers.
> >
> >Yes, there is a problem in that this data is sent from the target in random
> >order with respect to command sequence.  The SCSI command tag must be used
> >to associate data payload with the correct buffer.  This is not part of the
> >current TCP implementation.
> 
> Correct.  It is not part of a TCP implementation; it is part of iSCSI and 
> therefore independent.  Also, there are TCP implementations that do support 
> per connection buffer management (recall Solaris doing this a number of 
> years ago with fastbuf support as an example) so that is not new to the 
> implementation space and is clearly outside the TCP protocol definition and 
> workgroup area of concern.
> 
> 
> >You are describing something to do with out of sequence segment delivery
> >which is different from the target responding out of sequence to the
> >commands.  You start with an invalid assumption.  If you are discussing Zero
> >Copy TCP, then this operation does not include the extraction of the data
> >from the encapsulation, nor does it suffer from out of sequence TCP
> >segments.  You miss the point and why I include the term Content Directed
> >Placement.
> 
> A zero-copy TCP is a content directed placement of the data.  Content 
> directed placement implicitly implies header / data split including the 
> iSCSI protocol headers.  Don't see what the problem is.
> 
> 
> > > Now, when valid-out-order packets arrive TCP should initiate its error
> > > recovery algorithms as appropriate.  Implementations are also required to
> > > insure that the buffer completion event is not initiated until
> > > all packets
> > > have arrived.  From the application perspective it is completely
> > > opaque as
> > > to the order the buffer is filled in and is only focused on the
> > > completion
> > > event to know when it may examine the buffer's contents.
> >
> >You are confused as to the effort required to implement a means to direct
> >the encapsulated data to the buffer allocated within the SCSI request.  We
> >are not discussing Zero Copy TCP.  We are discussing Content Directed
> >Placement.  If it where not for this desire, there would be no reason to
> >discover the PDU with missing segments.
> 
> Still not an issue.  The PDU with the missing segment is recovered by TCP 
> and the iSCSI session does not complete any out-of-order PDUs until this 
> occurs. The buffers are therefore opaque to the application and the adapter 
> is kept thin and fast.
> 
> 
> > > I see nothing here that requires any API modifications to TCP-based
> > > applications nor any issue with interoperability or security since all of
> > > this activity is performed within the local receiving endnode.  I see no
> > > problem with a host-based implementation communicating with a TOE-based
> > > implementation since there are no wire protocol modifications
> > > w.r.t. TCP -
> > > the iSCSI headers are within TCP's data payload byte stream and are
> > > therefore independent of the TCP implementation itself.  What am
> > > I missing
> > > since you have brought up these issues on multiple occasions?
> >
> >If there was just a simple desire to do Zero Copy TCP, then sequence numbers
> >within TCP provides information needed to allow out of sequence processing
> >of the simple TCP stream.  This is NOT the desire in this case.  They wish
> >to extract data within the TCP stream as David Black described as a
> >(look-ahead) packet filter to deliver the SCSI data to the correct buffer
> >without the need to copy this data out of the TCP stream following a TCP
> >Zero Copy.  Even if you provided a true TCP Zero Copy, for SCSI there would
> >be one more additional copy required.  It would be the extraction of the
> >encapsulated data from stream and placement into the allocated SCSI buffers.
> >It makes no sense to be able to process SCSI PDUs out of sequence if this
> >were not the case.  This requires the exchange of buffer descriptors with
> >the network adapter that is not part of TCP today.
> 
> It is part of TCP in a variety of implementations.  All of this is 
> implementation specific and the techniques have been used for years.  The 
> fact that some implementations are not performance oriented is a separate 
> issue.  They can work by scanning and doing the copies if they desire or 
> they can get smart in the way they interact with a semi-smart (not even 
> fully off-loaded TCP/IP implementation) adapter and operate as many on this 
> reflector have advocated.  Again, I do not see any need to perform 
> additional copies as contended as this can be performed with some 
> intelligence in the buffer management and DMA algorithms.  Fairly 
> straight-forward and was done on other adapters supporting TCP back in 1992 
> on HP-UX and there has been numerous research and other implementations 
> that support similar algorithms.  Why is iSCSI so difficult if others can 
> do this for protocols such as NFS?
> 
> Mike
>