Re: More notes from Haifa

[Matt Wakeley <matt_wakeley@agilent.com>]

Costa Sapuntzakis wrote:

> On Fri, 30 Jun 2000, Matt Wakeley wrote:
>
> > > An optimization to iSCSI was discussed. The suggestion
> > > was that each TCP segment be the start of a new iSCSI PDU.
> > > This would be especially valuable for data transfers, as
> > > each segment would have enough information to place the data
> > > in memory at the remote end.
> >
> > You didn't note the results of this discussion.  I thought I remember
> > something that this could not be enforced upon TCP.  Something about it could
> > "send" a segment whenever it wants (thus that segment would not have a header)
> > and that the push bit wouldn't work either, because tcp could accept a few
> > more bytes after the push bit.
>
> Your recollections match mine. Thanks for filling in the gap.
>
> ---------------------------
>
> The motivation for the discussion are 1) to describe
> a simpler fast-path for iSCSI,  2) minimize data buffering needs in
> the presence of out-of-order reception.
>
> The bulk of the traffic on an iSCSI TCP connection will be SCSI data.
> SCSI data can be delivered to the buffer at each end in pretty much
> any order, just as long as it is all delivered. So, as an
> optimization, instead of keeping tons of SCSI data in a TCP receive
> queue, an iSCSI w/optimized TCP could parse out-of-order segments
> and deliver the SCSI data in them to the right SCSI buffer.

Yes - but only if there is a iSCSI header in the frame received.  If there is no
header, it can't be determined where to put the data. Yes, it's possible to detect
that this tcp segment is part of an iSCSI message *if* the header was previously
received by inspecting the length field in that header and placing the tcp segment
in the proper buffer.  But this does not cover *all* cases.  So, in the case where
the frame with the iSCSI header is lost (for example, a 64k data pdu), now all that
64K of data must be buffered and copied.  It would be *so* much easier if each tcp
segment contained an iSCSI header immediately after the TCP header, and each TCP
segment fit into exactly one ethernet frame.  This is basically how Fibre Channel
works.  A frame comes in, and from the information in the FC header, the payload can
be placed into memory.

There is also the issue of "getting out of sync".  I know that this "should not
happen" in a good implementation, but there are lots of people nervous about this
issue.

Also, as someone else posted, if every ethernet frame contained an iSCSI header, it
would be real easy to implement a protocol analyzer.  How is a protocol analyzer
supposed to figure out what's in a TCP stream?

These issues are a real detriment to the use of TCP as an iSCSI transport.  The
benefits of TCP are it's "reliable" delivery and congestion avoidance.  The
disadvantage is there is no "framing".

I hate to say it, but if another transport was utilized (say UDP), the same proven
algorithms that make TCPs reliablity and congestion control could be implemented in
the iSCSI layer along with the benefits of UDP framing.

(I wonder why all the proprietary start-ups seem to be using UDP...?)

TCP is great for things like telnet and email, but I'm less and less convinced it's
appropriate for mass storage.

> This would
> decrease the amount of buffering needed for TCP receive. In the
> limit, this optimization would decrease memory requirements by up to a
> factor of 2.
>
> As you point out, implementations can't rely on alignment in a TCP stream
> - to be interoperable they would have to implement unaligned parsing
> too. However, it would be reasonable to implement aligned parsing
> in the fast-path and kick unaligned parsing to the slow path.
>
> TCP stacks don't know the alignment requirements of applications
> because most applications fail to communicate message boundaries to the
> TCP stacks. Historically, this is because the TCP stack ignores
> message boundaries.
>
> However, it is possible to communicate message boundaries through
> the sockets interface! With the current TCP sockets interface, the sendmsg
> command, coupled with the MSG_EOR (end of record) flag, could be used to
> communicate iSCSI PDU boundaries to the TCP stack. The getopt
> command could be used to get the current path MTU of the connection.
>
> To communicate message boundaries over the wire, it
> would be expedient to add a bit to the TCP header saying:
> this segment is also the start of a new higher-layer PDU. This
> could help out-of-order parsing of the TCP stream.

Ok, but would this require a "change/enhancement" to the existing TCP standard?

>
>
> As for the PSH bit, RFC 1122, section 4.2.2.2, states "The PSH bit is not
> a record marker..."
>
> There are many constraints that can be envisioned
>
> 1) say nothing about alignment
> 2) iSCSI PDU headers do not span segments

With existing TCP implementations, how can this be enforced?


> 3) if more than one iSCSI PDU header appears in a segment, one
>    iSCSI PDU header always appears at the start of the segment

See #2.


> 4) zero or one iSCSI PDU header per segment and header always
>    aligned with start of segment.
>       - potentially creates lots of small segments

>
> 5) new segment always starts new iSCSI PDU
>       - implies iSCSI MTU <= TCP path MTU
>       - creates lots of small segments
>
> My preference is #3.

My preference is #5.  But the problem is comment to #2.

>
>
> -Costa

-Matt