Re: Out of order data delivery in response to R2T.

[julian_satran@il.ibm.com]

Barry,

You are right. It was I changed it by mistake while fixing some other text.

It reads again like in 03.

Regards,
Julo

"Barry Reinhold" <bbrtrebia@mediaone.net> on 27/03/2001 18:11:13

Please respond to "Barry Reinhold" <bbrtrebia@mediaone.net>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   "Jon Sreekanth" <jon.sreekanth@trebia.com>, "James Smart"
      <james.smart@trebia.com>
Subject:  Out of order data delivery in response to R2T.





Julian,
     Early versions of iSCSI allowed out of order responses to an R2T. This
was
fixed in rev 03 after some discussion. (See below) I have just noticed upon
review that it was taken out at 04 (see below) . Was this by error or
design. I have attached the email conversation related to that change for
your reference.

(Rev 03 - 2.16)
"An R2T MAY be answered with one or more iSCSI Data-out PDU with a
   matching Target Transfer Tag. If an R2T is answered with a single
   Data PDU the Buffer Offset in the Data PDU MUST be the same as the
   one specified by the R2T and the data length of the Data PDU must not
   exceed the Desired Data Length specified in R2T. If the R2T is
   answered with a sequence of Data PDUs the Buffer Offset and Length
   must be within the range of those specified by R2T, the last PDU
   should have the F bit set to 1, the Buffer Offsets and Lengths for
   consecutive PDUs SHOULD form a continuous non-overlapping range and
   the PDUs should be sent in increasing offset order."

(Rev 05 - 2.17)

"An R2T MAY be answered with one or more iSCSI Data-out PDU with a
   matching Target Transfer Tag. If an R2T is answered with a single
   Data PDU, the Buffer Offset in the Data PDU MUST be the same as the
   one specified by the R2T. The data length of the Data PDU must not
   exceed the Desired Data Length specified in R2T. If the R2T is
   answered with a sequence of Data PDUs, the Buffer Offset and Length
   MUST be within the range of those specified by R2T. The last PDU
   should have the F bit set to 1."

Email discussion of topic (follow up from San Diego meetings)

   Barry,

I can understand the rationale and the new text is strict (see attached).
However a bad digest can result in the need to redo part or the whole R2T
(or to reclaim all the data after the failure).

   If the R2T is answered with a sequence of Data PDUs the Buffer Offset
   and Length must be within the range of those
   specified by R2T, the last PDU should have the F bit set to 1, the
   Buffer Offsets and Lengths for consecutive PDUs SHOULD form a continuous
   non-overlapping range and the PDUs should be sent in increasing offset
   order.


   Regards,
   Julo

"Barry Reinhold" <bbrtrebia@mediaone.net> on 29/12/2000 23:48:30

Please respond to "Barry Reinhold" <bbrtrebia@mediaone.net>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   "Jon Sreekanth" <jon.sreekanth@trebia.com>, "James Smart"
      <james.smart@trebia.com>
Subject:  Out of order response to R2T




Julian,
     This is a bit of a delayed follow up to a conversation we had in San
Diego.
The issue has to do with how an initiator is allowed to respond to an R2T.

Right now the iSCSI draft says:

"If the R2T is
   answered with a sequence of Data PDUs the Buffer Offset and Length
   must be within the range of those
   specified by R2T and the last PDU should have the F bit set to 1; the
   Buffer Offsets and Lengths for consecutive PDUs SHOULD form a
   continuous range. "

Based on previous conversations this means that an initiator can break up
the delivery of the data into 4 segments and deliver the segments in any
order.

I would like to argue that this should be restricted such that when
responding to an R2T the data is delivered in order. I understand the logic
behind allowing the target to request data out of order based on the R2T.
This is consistent with Fibre Channel protocol and disk drive needs.
However, it is not clear to me why we should allow the iSCSI data PDUs sent
in response to the R2T to be out of order. I have three observations on
this:

1. The concept behind the target sending the R2T (based on analogy to the
FCP_XFER_RDY) is that the target is ready to receive data starting at
"offset" of a given length. This does not happen when the data is delivered
out of order forcing the end device to reassemble the information and
making
the check process more complicated.

2. This is specifically diallowed by Fibre Channel. Fibre Channel requires
that the data be delivered in one IU (sequence) and that the first frame of
the sequence must have the data offset set to the value sent in the
FCP_XFER_RDY frame. All the rest of the frames in the sequence must deliver
data in order by FC-FS rules.

For reference - FCP-2 clause 9.3

"If an FCP_XFER_RDY IU
is used to describe a data transfer and the first frame of the requested
FCP_DATA IU has a relative offset that
differs from the value in the FCP_DATA_RO field of the FCP_XFER_RDY IU, the
target shall post the error code


ôFCP_DATA Parameter mismatch with FCP_DATA_RO


ö in the FCP_RSP_INFO field of
the FCP_RSP IU."

I contacted Bob Snively (author of FCP-2) to confirm this. Bob is willing
to
discuss the semantics of the FCP_XFER_RDY data transfer with you if you
wish
to pick up that thread.

3. We have not seen this behavior in the field. If some device actually
took
advantage of this flexibility I suspect a number of implementation issues
would come up. In my experience these types of options hinder
interoperabiliy.

Also note that if a device is going to translate from iSCSI to FC, it is
going to have a difficult time with this. It must buffer up all the iSCSI
frames until it finds the first piece. It the data is sent in reverse order
and the transfer is large the buffering could be significant. Testing this
is a difficult process, and when testing is difficult interoperability
problems creep in.

In summary I would like to suggest that out of order data delivery in
response to an R2T is not a helpful feature, hinders ineroperability, is
not
compatible with Fibre Channel, and therefore makes interconnecting with
Fibre Channel legacy devices more difficult. I would like to see us adopt
the same semantics for iSCSI in this requard as Fibre Channel has.

Thanks for your time on this,

Barry Reinhold


Barry Reinhold
Principal Architect
Trebia Networks
barry.reinhold@trebia.com
603-868-5144/603-659-0885/978-929-0830 x138