RE: iSCSI: Markers

["Julian Satran" <Julian_Satran@il.ibm.com>]

Very good point. That is what we had in mind when we suggested COWS.

Julo

"Somesh Gupta" <somesh_gupta@silverbacksystems.com> Sent by: owner-ips@ece.cmu.edu 13-01-02 01:06 Please respond to somesh_gupta

To:        John Hufferd/San Jose/IBM@IBMUS, "Stuart Cheshire" <cheshire@apple.com>         cc:        <ips@ece.cmu.edu>         Subject:        RE: iSCSI: Markers

It is important to realize that there is compatibility between
COWS with TCP implementation changes and COWS without TCP
implementation changes.

Sender with COWS & TCP implementation changes works fine
with a receiver with COWS but without TCP implementation
changes.

Sender with COWS & without TCP implementation changes works
fine with a receiver with COWS and which is implemented
to take advantage of but work with a sender that is not
updated.

Sender with COWS & TCP implementation changes works best
with a receiver with COWS and takes advantage of alignment.

What this says is that COWS preceding layer 5 header is
a very solid migration path towards the most desirable
objective of framing. There should be an iSCSI option to
indicate whether the sender TCP has been changed to
support framing or not. Over time as more and more sender
TCP implementations can evolve to be framing compliant.

Of course, all the issues that have been raised are
still valid, and it is probably best to not mandate
anything at this point.

Somesh


> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> John Hufferd
> Sent: Saturday, January 12, 2002 3:18 AM
> To: Stuart Cheshire
> Cc: ips@ece.cmu.edu
> Subject: Re: iSCSI: Markers
>
>
>
> Stuart,
> thanks for the contribution.
>
> I found your views interesting, however, in the last part of your message,
> when you attempted to contrast it to FIM, you mixed a Framing discussion
> with a marker discussion.  Here is what I mean:
>
> You made much of your arguments to address COWS as being part of a Framing
> (Forcing Segmentation alignment with PDUs).  Many of us are already a fan
> of Framing, and understand that both the Key+length and COWS have an
> important debate to hold with lots of real world ASIC Vendor Input needed.
>
> But then you jumped to contrasting COWS with Framing vrs FIM.
> That has not
> been the focus of the discussion.  I can not think of anyone that is
> suggesting FIM in place of any Framing approach.  Framing is the goal, and
> honest debate on the Type, will be needed and be useful.  The debate then
> occurs in two areas, what do we do until we have a Framing solution
> (Framing being out of the domain of iSCSI, so we must wait until one is
> accepted before we can point to it.), and what kind of a Framing solution
> should we wish, so that we can influence the Framing debate.
> Hence the Six
> Options.
>
> So some of your discussions on COWS, is completely valid for Framing, but
> is not quite as useful, without framing. When it comes to not having
> Framing (with Segmentation starts, etc) some will argue that COWS is much
> more complex, and much higher overhead.
>
> So again, thank you for your contribution, I just wanted to correct that
> slight misalignment at the end of your note.
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: hufferd@us.ibm.com
>
>
> Stuart Cheshire <cheshire@apple.com> on 01/11/2002 05:39:17 PM
>
> To:    John Hufferd/San Jose/IBM@IBMUS, <ips@ece.cmu.edu>
> cc:
> Subject:    Re: iSCSI: Markers
>
>
>
> I'm new to this list, so I should introduce myself.
>
> My name is Stuart Cheshire; I'm the author of Consistent Overhead Byte
> Stuffing (COBS), the framing technique from which COWS derives. I'm not
> working on any iSCSI product, but if COBS can contribute to iSCSI, then
> I'm happy to offer a little of my time, as much as I can spare, to help
> clarify what COWS does and does not do, to help people make an informed
> decision whether or not COWS is the right solution for iSCSI.
>
> ----
>
> Assumption: A high-performance receiver is harder than a high-performance
> sender.
>
> This is because the sender is in control. It knows where the data is
> coming from in memory, and where it is going to on the network. The
> sending host knows and can control all aspects of the communication: what
> order iSCSI messages are delivered onto the wire, how big each one is,
> and at what time they are sent. If the sender wants to do some kind of
> housekeeping that prevents it from sending packets for a few
> milliseconds, then it has the option of doing that without terrible
> consequences.
>
> The receiver has a much harder time. It never knows what packet is going
> to arrive next, or how big it will be, or where it will be from, or where
> it will have to go to in memory. Packet loss/corruption/reordering makes
> things even more unpredictable. A receiver doesn't have the luxury of
> being able to not receive packets for a few milliseconds if it is busy
> with something else.
>
> For this reason, it makes sense to see what the sender can do to make the
> receiver's life a little easier. If the receiver could receive each TCP
> segment and process it in isolation, determining where to place it in
> memory solely from information within that TCP segment, without reference
> to data from other TCP segments (which may not have arrived yet), then it
> would be easier to make a high-performance receiver.
>
> What can we do to enable independent segment processing and idempotent
> direct data placement at the receiver?
>
> My first choice would be to add a couple of extra bits to the TCP header;
> a "start of message" bit and an "end of message" bit. The "start of
> message" bit indicates that the first byte of TCP data in the segment is
> also the first byte of an iSCSI message; the "end of message" bit
> indicates that the last byte of TCP data in the segment is also the last
> byte of an iSCSI message. When a receiver receives a TCP segment with
> both bits set, it knows with certainty that it has one (or more) complete
> iSCSI messages in the TCP segment and can immediately decode enough of
> the iSCSI message header(s) to determine where in memory to place the
> data.
>
> Unfortunately, adding extra bits to the TCP header is not viable. From a
> political point of view, trying to change the TCP on-the-wire protocol is
> a non-starter. From a practical point of view, there are too many routers
> and firewalls and similar devices that will throw away TCP packets with
> bits they don't understand.
>
> Given that out-of-band framing using header bits is not possible, the
> alternative is in-band framing using only information in the TCP data
> stream itself.
>
> If we can design our sender to normally send exactly one iSCSI message
> per TCP segment, and we have a way for our receiver to reliably verify
> that the received TCP segment contains exactly one iSCSI message, then
> the receiver can implement idempotent direct data placement for each TCP
> segment as it is received, without reference to state from previous TCP
> segments on that connection (which may not have arrived yet).
>
> The problem left to solve is how the receiver can reliably verify that
> the received TCP segment contains exactly one iSCSI message. It can do
> this by checking to see whether the TCP segment data begins with some
> special marker pattern, as long as it knows that this special marker
> pattern cannot appear anywhere within the body of valid iSCSI message
> data. This necessarily entails processing ("stuffing") the body of the
> iSCSI message to eliminate inadvertent occurrences of the special marker
> pattern before sending, and then reversing this transformation to restore
> the original data after reception.
>
> If the receiver finds that the segment does not begin with the special
> marker pattern, then it knows that the sender segmentation has not been
> maintained (or it is talking to an old TCP sender that doesn't support
> sender segmentation) and it has to fall back to treating the TCP data
> stream as a raw unstructured byte stream, with message boundaries
> indicated by occurrences of the the marker pattern. The important thing
> is that the receiver still works correctly, even though the performance
> will be lower.
>
> This prefer-sender-segmentation-but-verify approach is important. If the
> outgoing data is not processed to guarantee that the special marker
> pattern cannot occur, then malicious users might be able to subvert the
> protocol by putting contrived patterns in their data. Remember the days
> where you could make a user's modem hang up by sending them an email
> containing the text "+++ATH"? (Apologies to anyone reading this via modem
> who just had their telephone line hang up.)
>
> Another benefit of using in-band framing like this is that we can deploy
> it immediately using unmodified TCP stacks. In the future we can use
> enhanced sender TCP implementations that take steps to maintain segment
> boundaries, and smart receivers will get a performance boost from that,
> but it is a compatible upgrade that changes only the implementation, not
> the on-the-wire protocol.
>
> Of course, we don't get anything for free. If we want to receiver to be
> able to determine with 100% certainty that it has received a complete
> iSCSI message in one TCP segment, then the sender will have to do some
> work to enable that. This is the cost of COWS. It gives 100% framing
> certainty, but at the cost of checking the outgoing data for inadvertent
> occurrences of the special marker pattern, and eliminating them. There's
> no way for a sender to tell whether the outgoing data contains
> inadvertent occurrences of the special marker pattern if the sender is
> not willing to look at the data.
>
> On the plus side, the cost of COWS encoding is modest compared to some
> alternatives. COWS-encoding adds a little header but otherwise doesn't
> change the size of the outgoing data, ever. No matter how many
> occurrences of the framing marker pattern are found, the encoded output
> length is always exactly the same: the length of the input plus the
> length of the fixed-size framing header (typically two words). If the
> framing marker pattern is chosen to be something that is rare in normal
> (non-malicious) data, then in the common-case the encoding step will be a
> read-only operation: scan the data, determine that it contains no framing
> markers, set the COWS header to indicate that the data contains no
> framing markers, and send it.
>
> In contrast, when using Fixed Interval Markers, if a marker happens to
> fall in the middle of the data you are sending, then it creates a 'hole'
> in the middle of data that used to be contiguous, and the block of
> outgoing data changes size. On the receiving side, the 'hole' created by
> the marker has to be repaired in the process of transferring the data
> into memory. When using Fixed Interval Markers, when a receiver gets a
> TCP segment that contains no marker, it cannot reliably determine what it
> is supposed to do with that segment (where to put it in memory) without
> referring the state from the previous TCP segments of that connection. I
> don't believe that FIM can provide efficient idempotent direct data
> placement for inbound TCP segments, because you can't rely on any given
> received segment containing a marker via which the receiver can verify
> that the segment contains a complete iSCSI message.
>
> In summary:
>
> My first choice would be to modify the TCP protocol to support
> preservation of upper-level message boundaries.
>
> Given that this is not possible, I think COWS provdes a good alternative.
>
> Stuart Cheshire <cheshire@apple.com>
>  * Wizard Without Portfolio, Apple Computer
>  * Chairman, IETF ZEROCONF
>  * www.stuartcheshire.org
>
>
>
>
>
>