RE: ISCSI: Immediate data extending beyond a single PDU

["Barry Reinhold" <bbrtrebia@mediaone.net>]

Julian,
	I  think one could carry on a debate about what the draft currently states
about the coupling of immediate and unsolicited data - but it is probably
more productive to figure out if we want to have both the concept of
immediate data and unsolicited data and then worry about the wording in the
draft.
	I am of the opinion that immediate data (data within a single PDU) is a
useful distincition from unsolicited data (iSCSI DATA PDU sent without R2T).
The reasoning behind this is that there are a number of write IOs that can
be satisfied by providing a buffer of 8K or less. If this data is contained
in the command PDU the context can be created for the command, and then the
data just happens to be sitting there with the context to process it. Thus
the target can send the iSCSI response PDU without creating any special
context or mechansims - should have nice low latency characteristics for
small writes.
	For devices that don't want to use the initiator task tag for establishing
context of an iSCSI DATA PDU, immediate data is a nice thing. An iSCSI DATA
PDU without a target task tag may not be such a nice thing.
	Question here - What would you lose if you supported only immediate data?
It seems like this meets some of the common application requirements that I
am aware of and is friendly to what I think is the more common approach to
target processing of DATA PDUs.

>-----Original Message-----
>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
>julian_satran@il.ibm.com
>Sent: Sunday, February 04, 2001 5:20 AM
>To: ips@ece.cmu.edu
>Subject: RE: ISCSI: Immediate data extending beyond a single PDU
>
>
>
>
>Barry,
>
>I think that the current draft does not allow you to accept immediate but
>not unsolicited.
>You can either allow unsolicited (including immediate) or request always
>R2T.
>
>And yes - if the expected length is larger that what you sent as
>unsolicited (even if the unsolicited was less than the limit) the target is
>supposed to send R2T (there is only one unsolicited burst per command).
>
>The limit for immediate data is implicit - the max-PDU length and is
>less-then-or-equal the first burst.
>
>Your proposal is interesting but I doubt that it is very useful.
>
>Unsolicited data help avoid an RTT on writes but require resources to be
>allocated (that is why a target may not accept them).
>
>Immediate data is an "optimized" form of unsolicited data in which the
>command and data are collapsed to one unit. Nevertheless the immediate data
>are unsolicited and require resources to be allocated ahead of time. If you
>have larger amounts to send than what fits into one PDU the "collapse gain"
>is minimal and does not justify the added complexity of 2 options.
>
>Regards,
>Julo
>
>
>
>"Barry Reinhold" <bbrtrebia@mediaone.net> on 02/02/2001 21:06:29
>
>Please respond to "Barry Reinhold" <bbrtrebia@mediaone.net>
>
>To:   Julian Satran/Haifa/IBM@IBMIL
>cc:
>Subject:  RE: ISCSI: Immediate data extending beyond a single PDU
>
>
>
>
>Julian,
>     I agree with the conceptual idea of immediate data, but further
>refinement
>is required. As a receiver, if I have allowed immediate data, but not
>unsolicited data, and I get a write command with an expected data transfer
>length that is greater then the amount of data in the command PDU, do I
>send
>an R2T or not (or from the sender's point of view does he wait for an R2T
>to
>complete the data transfer?) The draft is not clear on this, yet it is an
>externally visible behavior that needs to be defined for interoperability.
>
>I believe that what we need here is a refinement of two distinct concepts,
>that of immediate data and that of unsolicited data. Here is my take at it:
>
>1. Immediate data - Data that is sent within a single iSCSI SCSI command
>PDU. Immediate data shall be fully contained within a single iSCSI command
>PDU, and shall be limitted by the negotiated value of ImmediateData. The
>size of Immediate data is not subject to the limits of FirstBurst.
>
>2. Unsolicited data - Data that may be transfered to a target without an
>explicit R2T being sent to request its transfer. Unsolicited data is sent
>in
>iSCSI SCSI data PDUs in which the target transfer tag is invalid. The
>maximum amount of unsolicited data is negotiated between the initiator and
>target through the FirstBurst key. If unsolicited data is supported by the
>target, upon receipt of a write command, the target shall not send an R2T
>until an iSCSI Data PDU with the Final bit set has been received. After
>reception of an iSCSI Data PDU with the Final bit set R2Ts shall be sent by
>the target to request transmission of any additional data associated with
>this command. The initiator shall not send any additional unsolicited data
>for this command.
>
>Interaction between Immediate data and Unsolicited data -
>
>Immediate data and Unsolicited data are features that may be selected
>independently of one another. Support of either is optional.
>
>Note: Based on the current specification of FirstBurst, support for
>unsolicited data is not optional. One must select between a range of
>1-65536. I think we would want to make this optional and indicate non
>support for unsolicited data by setting FirstBurst to 0.
>
>
>>-----Original Message-----
>>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
>>julian_satran@il.ibm.com
>>Sent: Friday, February 02, 2001 6:41 AM
>>To: ips@ece.cmu.edu
>>Subject: Re: ISCSI: Immediate data extending beyond a single PDU
>>
>>
>>
>>
>>Barry,
>>
>>There is a misunderstanding:
>>
>>You can send EITHER immediate data OR a sequence of unsolicited PDUs.
>>
>>Immediate data was conceived for short writes - for which an initiator
>does
>>not want to do
>>two socket operations or 2 HBA operations.
>>
>>Larger unsolicited bursts can be sent with a sequence and then we felt
>that
>>you won't gain too much by having the first one glued to the command.
>>
>>
>>Julo
>>
>>"Barry Reinhold - Lamprey" <bbr@lampreyNetworks.com> on 01/02/2001
>22:04:28
>>
>>Please respond to "Barry Reinhold - Lamprey" <bbr@lampreyNetworks.com>
>>
>>To:   Julian Satran/Haifa/IBM@IBMIL
>>cc:   "ISCSI" <ips@ece.cmu.edu>
>>Subject:  ISCSI: Immediate data extending beyond a single PDU
>>
>>
>>
>>
>>Julian,
>>
>>My understanding of your email in conjunction with the current draft is:
>>
>>The initiator may send zero or more immediate data bytes in the CMD PDU,
>>and
>>may send additional unsolicited DATA PDUs with further immediate data,
>with
>>the
>>last PDU having the F bit set. The total amount of immediate data to be
>>limited
>>by the value negotiated at login.
>>
>>There are two issues I have with this:
>>
>>First, the target does not know how much immediate data has being sent.
>>There
>>is no indication in the CMD frame to indicate that more immediate data is
>>to
>>follow (whether or not the CMD frame contains any immediate data). The
>>initiator is free to send any amount, from zero up to the negotiated
>value.
>>Therefore, the target will likely send an R2T to the host at whatever
>>boundry
>>it believes is present - which may cause a retransmission of much or all
>of
>>the
>>immediate data.
>>
>>Second, the additional unsolicited PDU's do not contain a target
>>task/transfer
>>tag. As such, in the mainline processing, you are perhaps forcing the
>>target
>>to
>>look up the Initiator Task Tag in its list of active i/o's. This may exact
>>a
>>heavy performance penalty as optimizing this lookup maybe
>cost-prohibitive.
>>The
>>only other scenarios in which the target had to perform this lookup was in
>>low-occurance error/abort conditions.
>>
>>Are there negative implications to keeping immediate data to a single PDU?
>>
>>-- Barry
>>
>>
>>julian_satran@il.ibm.com wrote:
>>
>>> Barry,
>>>
>>> Immediate data is "attached to the command". If your unsolicited data
>>(the
>>> so called initial burst) is larger that what you are ready to commit to
>a
>>> single PDU you can send several PDUs with the  last having the F bit.
>>After
>>> that the target can decide that it wants more and send an R2T or send
>>> status.
>>>
>>> You are no supposed to send both immediate and separate PDU.
>>>
>>> I admit that the text is bit confusing (in 1.2.5)  and there is an error
>>> also in the appendix.
>>> 2.7.1 talk about unsolicited data (not immediate) (the context is data
>>> PDUs).
>>>
>>> The new version of 1.2.5 now reads:
>>>
>>>    Unsolicited data can be sent as part of an iSCSI command PDU
>>("immediate
>>>    data") or an in separate iSCSI data PDUs.  An initiator may send
>>>    unsolicited data either as immediate (up to the negotiated maximum
>PDU
>>>    size - DataPDULength - disconnect-reconnect mode page) or in a
>>separate
>>>    PDU sequence (up to the negotiated limit - FirstBurstSize -
>>>    disconnect-reconnect mode page). All subsequent data have to be
>>>    solicited.  The maximum size of an individual data PDU or the
>>>    immediate-part of the initial unsolicited burst as well as the
>initial
>>>    burst size MAY be negotiated at login.
>>>
>>> Thanks,
>>> Julo
>>>
>>> "Barry Reinhold - Lamprey" <bbr@lampreyNetworks.com> on 29/01/2001
>>16:34:30
>>>
>>> Please respond to "Barry Reinhold - Lamprey" <bbr@lampreyNetworks.com>
>>>
>>> To:   Julian Satran/Haifa/IBM@IBMIL
>>> cc:   "Jon Sreekanth" <jon.sreekanth@trebia.com>, "James Smart"
>>>       <james.smart@trebia.com>
>>> Subject:
>>>
>>> Issue:
>>>
>>> I am unclear on how to respond to an iSCSI SCSI command PDU when there
>is
>>> immediate data but not enough to meet the requirements of the IO
>>operation
>>> as given in the expected data transfer length field. Should an R2T be
>>> issued
>>> or not?
>>>
>>> Background:
>>>
>>> Clause 2.7.1 in 03 reads:
>>>
>>> "2.7.1 F (Final) bit
>>>
>>>    This bit is 1 for the last PDU of immediate data or the last PDU of a
>>>    sequence answering a R2T."
>>>
>>> This suggests that immediate data can extend beyond the iSCSI SCSI CMD
>>PDU
>>> (there is no final bit on the command PDU)but the closet thing we have
>to
>>a
>>> definition for immediate data, which is in clause 1.2.5 below, suggests
>>> that
>>> immediate data is limitted to the command PDU:
>>>
>>> "Outgoing SCSI data (initiator to target - user data or command
>>>    parameters) will be sent as either solicited data or unsolicited
>>>    data.  Solicited data are sent in response to Ready To Transfer (R2T)
>>>    PDUs. Unsolicited data can be part of an iSCSI command PDU
>>>    ("immediate data") or an iSCSI data PDU.  An initiator may send
>>>    unsolicited data (immediate or in a separate PDU) up to the
>>>    negotiated limit (initial burst size - mode page 02h). All subsequent
>>>    data have to be solicited.  The maximum size of an individual data
>>>    PDU or the immediate-part of the initial unsolicited burst MAY be
>>>    negotiated at login (maximum connect size - mode page 02h)."
>>>
>>> Unsolicited data is bounded in a number of ways, but most significant to
>>> this issue is in the description of the login key useR2t which states:
>>>
>>> "Note than only the first
>>>    outgoing data item (either immediate data or a separate PDU) can be
>>>    sent unsolicited by a R2T."
>>>
>>> Given the above it is unclear to me if the concept of immediate data is:
>>>
>>> 1. Write data that can be sent without an R2T (unsolicited write
>data)and
>>> starts in the command PDU.
>>> 2. Write data that is entirely contained within the command PDU. (my
>>> initial
>>> concept of immediate data)
>>> 3. The non-zero portion of data, in a possible multi-PDU write
>operation,
>>> that is contained in the iSCSI SCSI command PDU. Where the remaining
>PDUs
>>> of
>>> the write operation must be solicited with an R2T.
>>>
>>> Given the current definition of the Final bit and the current limits on
>>> unsolicited data it is unclear to me what a target should do when
>>receiving
>>> an iSCSI SCSI command PDU that has immediate data in it, but not
>>sufficient
>>> data to meet the expected data transfer length specificied in the
>>command.
>>> Does an R2T go out of not?
>>>
>>> Barry Reinhold
>>> 603-659-0885
>>
>>Barry Reinhold
>>603-659-0885
>>
>>
>>
>>
>
>
>
>