RE: Framing Discussion

[Douglas Otis <dotis@sanlight.net>]

Michael,

> At 10:55 AM 12/20/2000 -0800, Douglas Otis wrote:
> >JP,
> >
> >The VM page flipping will require blocks to be greater than or equal to
> >pages in size and that these blocks are aligned at page
> boundaries.  Neither
> >of these assumptions are true. The goal is clear, the NIC will
> examine the
>
> It is more accurate to state that blocks are not always aligned since
> clearly one can make them aligned in a given implementation and
> quite often
> they are.

Aligned to what?  With iSCSI there is a PDU header that must be stripped.  I
assume you expect this header to vanish?

> >content and then direct data payload to the system through the NIC
> >interface.  The desire is to keep the buffers on the NIC small and thus
> >allow out of sequence processing of the TCP stream.  This must
> be seen as a
> >modification to the normal TCP implementation.  Such operation should
> >include a complete description of the API to allow consideration of NIC
> >design, inter-operability and security requirements.
>
> Need to separate out completion and ACK generation semantics from how and
> the order that data is DMA'ed into the target buffers.

Yes, there is a problem in that this data is sent from the target in random
order with respect to command sequence.  The SCSI command tag must be used
to associate data payload with the correct buffer.  This is not part of the
current TCP implementation.

> If the buffer is
> known as the target for a given operation, then the buffer can be written
> to in any order that is desired - this true for all I/O implementations
> today.  If valid out-of-order packets (critical to ensure data
> integrity on
> a per header / packet / segment basis since data cannot be DMA'ed
> until one
> is sure it is valid) arrive and it can be determined that the payload of
> these packets belongs to a known target buffer, the data can be
> DMA'ed into
> that buffer at the proper offset.  There is nothing in TCP to
> preclude such
> a design and in fact this type of operation is implemented by
> some vendors
> today as part of their per connection copy avoidance solutions.

You are describing something to do with out of sequence segment delivery
which is different from the target responding out of sequence to the
commands.  You start with an invalid assumption.  If you are discussing Zero
Copy TCP, then this operation does not include the extraction of the data
from the encapsulation, nor does it suffer from out of sequence TCP
segments.  You miss the point and why I include the term Content Directed
Placement.

> Now, when valid-out-order packets arrive TCP should initiate its error
> recovery algorithms as appropriate.  Implementations are also required to
> insure that the buffer completion event is not initiated until
> all packets
> have arrived.  From the application perspective it is completely
> opaque as
> to the order the buffer is filled in and is only focused on the
> completion
> event to know when it may examine the buffer's contents.

You are confused as to the effort required to implement a means to direct
the encapsulated data to the buffer allocated within the SCSI request.  We
are not discussing Zero Copy TCP.  We are discussing Content Directed
Placement.  If it where not for this desire, there would be no reason to
discover the PDU with missing segments.

> I see nothing here that requires any API modifications to TCP-based
> applications nor any issue with interoperability or security since all of
> this activity is performed within the local receiving endnode.  I see no
> problem with a host-based implementation communicating with a TOE-based
> implementation since there are no wire protocol modifications
> w.r.t. TCP -
> the iSCSI headers are within TCP's data payload byte stream and are
> therefore independent of the TCP implementation itself.  What am
> I missing
> since you have brought up these issues on multiple occasions?

If there was just a simple desire to do Zero Copy TCP, then sequence numbers
within TCP provides information needed to allow out of sequence processing
of the simple TCP stream.  This is NOT the desire in this case.  They wish
to extract data within the TCP stream as David Black described as a
(look-ahead) packet filter to deliver the SCSI data to the correct buffer
without the need to copy this data out of the TCP stream following a TCP
Zero Copy.  Even if you provided a true TCP Zero Copy, for SCSI there would
be one more additional copy required.  It would be the extraction of the
encapsulated data from stream and placement into the allocated SCSI buffers.
It makes no sense to be able to process SCSI PDUs out of sequence if this
were not the case.  This requires the exchange of buffer descriptors with
the network adapter that is not part of TCP today.

Doug

> Mike