RE: iSCSI: Markers

["Somesh Gupta" <somesh_gupta@silverbacksystems.com>]

Stuart,

Are there any IPR issues associated with COBS that you might
want to share i.e. patents etc?

Somesh

> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Stuart Cheshire
> Sent: Friday, January 11, 2002 5:39 PM
> To: John Hufferd; ips@ece.cmu.edu
> Subject: Re: iSCSI: Markers
>
>
> I'm new to this list, so I should introduce myself.
>
> My name is Stuart Cheshire; I'm the author of Consistent Overhead Byte
> Stuffing (COBS), the framing technique from which COWS derives. I'm not
> working on any iSCSI product, but if COBS can contribute to iSCSI, then
> I'm happy to offer a little of my time, as much as I can spare, to help
> clarify what COWS does and does not do, to help people make an informed
> decision whether or not COWS is the right solution for iSCSI.
>
> ----
>
> Assumption: A high-performance receiver is harder than a high-performance
> sender.
>
> This is because the sender is in control. It knows where the data is
> coming from in memory, and where it is going to on the network. The
> sending host knows and can control all aspects of the communication: what
> order iSCSI messages are delivered onto the wire, how big each one is,
> and at what time they are sent. If the sender wants to do some kind of
> housekeeping that prevents it from sending packets for a few
> milliseconds, then it has the option of doing that without terrible
> consequences.
>
> The receiver has a much harder time. It never knows what packet is going
> to arrive next, or how big it will be, or where it will be from, or where
> it will have to go to in memory. Packet loss/corruption/reordering makes
> things even more unpredictable. A receiver doesn't have the luxury of
> being able to not receive packets for a few milliseconds if it is busy
> with something else.
>
> For this reason, it makes sense to see what the sender can do to make the
> receiver's life a little easier. If the receiver could receive each TCP
> segment and process it in isolation, determining where to place it in
> memory solely from information within that TCP segment, without reference
> to data from other TCP segments (which may not have arrived yet), then it
> would be easier to make a high-performance receiver.
>
> What can we do to enable independent segment processing and idempotent
> direct data placement at the receiver?
>
> My first choice would be to add a couple of extra bits to the TCP header;
> a "start of message" bit and an "end of message" bit. The "start of
> message" bit indicates that the first byte of TCP data in the segment is
> also the first byte of an iSCSI message; the "end of message" bit
> indicates that the last byte of TCP data in the segment is also the last
> byte of an iSCSI message. When a receiver receives a TCP segment with
> both bits set, it knows with certainty that it has one (or more) complete
> iSCSI messages in the TCP segment and can immediately decode enough of
> the iSCSI message header(s) to determine where in memory to place the
> data.
>
> Unfortunately, adding extra bits to the TCP header is not viable. From a
> political point of view, trying to change the TCP on-the-wire protocol is
> a non-starter. From a practical point of view, there are too many routers
> and firewalls and similar devices that will throw away TCP packets with
> bits they don't understand.
>
> Given that out-of-band framing using header bits is not possible, the
> alternative is in-band framing using only information in the TCP data
> stream itself.
>
> If we can design our sender to normally send exactly one iSCSI message
> per TCP segment, and we have a way for our receiver to reliably verify
> that the received TCP segment contains exactly one iSCSI message, then
> the receiver can implement idempotent direct data placement for each TCP
> segment as it is received, without reference to state from previous TCP
> segments on that connection (which may not have arrived yet).
>
> The problem left to solve is how the receiver can reliably verify that
> the received TCP segment contains exactly one iSCSI message. It can do
> this by checking to see whether the TCP segment data begins with some
> special marker pattern, as long as it knows that this special marker
> pattern cannot appear anywhere within the body of valid iSCSI message
> data. This necessarily entails processing ("stuffing") the body of the
> iSCSI message to eliminate inadvertent occurrences of the special marker
> pattern before sending, and then reversing this transformation to restore
> the original data after reception.
>
> If the receiver finds that the segment does not begin with the special
> marker pattern, then it knows that the sender segmentation has not been
> maintained (or it is talking to an old TCP sender that doesn't support
> sender segmentation) and it has to fall back to treating the TCP data
> stream as a raw unstructured byte stream, with message boundaries
> indicated by occurrences of the the marker pattern. The important thing
> is that the receiver still works correctly, even though the performance
> will be lower.
>
> This prefer-sender-segmentation-but-verify approach is important. If the
> outgoing data is not processed to guarantee that the special marker
> pattern cannot occur, then malicious users might be able to subvert the
> protocol by putting contrived patterns in their data. Remember the days
> where you could make a user's modem hang up by sending them an email
> containing the text "+++ATH"? (Apologies to anyone reading this via modem
> who just had their telephone line hang up.)
>
> Another benefit of using in-band framing like this is that we can deploy
> it immediately using unmodified TCP stacks. In the future we can use
> enhanced sender TCP implementations that take steps to maintain segment
> boundaries, and smart receivers will get a performance boost from that,
> but it is a compatible upgrade that changes only the implementation, not
> the on-the-wire protocol.
>
> Of course, we don't get anything for free. If we want to receiver to be
> able to determine with 100% certainty that it has received a complete
> iSCSI message in one TCP segment, then the sender will have to do some
> work to enable that. This is the cost of COWS. It gives 100% framing
> certainty, but at the cost of checking the outgoing data for inadvertent
> occurrences of the special marker pattern, and eliminating them. There's
> no way for a sender to tell whether the outgoing data contains
> inadvertent occurrences of the special marker pattern if the sender is
> not willing to look at the data.
>
> On the plus side, the cost of COWS encoding is modest compared to some
> alternatives. COWS-encoding adds a little header but otherwise doesn't
> change the size of the outgoing data, ever. No matter how many
> occurrences of the framing marker pattern are found, the encoded output
> length is always exactly the same: the length of the input plus the
> length of the fixed-size framing header (typically two words). If the
> framing marker pattern is chosen to be something that is rare in normal
> (non-malicious) data, then in the common-case the encoding step will be a
> read-only operation: scan the data, determine that it contains no framing
> markers, set the COWS header to indicate that the data contains no
> framing markers, and send it.
>
> In contrast, when using Fixed Interval Markers, if a marker happens to
> fall in the middle of the data you are sending, then it creates a 'hole'
> in the middle of data that used to be contiguous, and the block of
> outgoing data changes size. On the receiving side, the 'hole' created by
> the marker has to be repaired in the process of transferring the data
> into memory. When using Fixed Interval Markers, when a receiver gets a
> TCP segment that contains no marker, it cannot reliably determine what it
> is supposed to do with that segment (where to put it in memory) without
> referring the state from the previous TCP segments of that connection. I
> don't believe that FIM can provide efficient idempotent direct data
> placement for inbound TCP segments, because you can't rely on any given
> received segment containing a marker via which the receiver can verify
> that the segment contains a complete iSCSI message.
>
> In summary:
>
> My first choice would be to modify the TCP protocol to support
> preservation of upper-level message boundaries.
>
> Given that this is not possible, I think COWS provdes a good alternative.
>
> Stuart Cheshire <cheshire@apple.com>
>  * Wizard Without Portfolio, Apple Computer
>  * Chairman, IETF ZEROCONF
>  * www.stuartcheshire.org
>
>
>