Re: iSCSI: SNACK R2T/Data

[Sandeep Joshi <sandeepj@research.bell-labs.com>]

The ack effect could also be achieved using SCSI Command Retry (use
the SCSI Command's expDataSN field)  The target stops sending 
Read PDUs if it runs out of buffers, and the initiator then
times out and retries the command.   This is what a target
would have to do in the absence of some data ack.

I looked at the archived emails on this subject (when dataSN
was called DataRN)  The Orlando minutes also have a brief reference.
I suspect the atmosphere then was that the recovery scheme was 
still partly defined and this DataSN ack scheme was adding to 
the general sense of complexity (..so lets dispose it)

I agree with Julian & Matthew that some form of data acks would
add value for long transfers.  The acks will be cumulative and
not per ReadData PDU (based on prenegotiated intervals or on 
a "send_ack" bit in the ReadData PDU set by the target) 

Btw, I see that the expDataSN field has disappeared from the
SCSI Command PDU.   The Task Management Command also does not
have anything related to expDataSN.  I may have missed an
email on this ?

-Sandeep

Julian Satran wrote:
> 
> It does not have to involve a net  type but it must be a specific request
> (like SNACK) as there might be no output
> traffic at the time. The overhead involved is minor (as it can be done only
> once per sequence and be cumulative).
> 
> Julo
> 
> 
>                     Michael Schoberg
>                     <michael_schober       To:     "'ips@ece.cmu.edu'" <ips@ece.cmu.edu>
>                     g@cnt.com>             cc:
>                     Sent by:               Subject:     RE: iSCSI: SNACK R2T/Data
>                     owner-ips@ece.cm
>                     u.edu
> 
> 
>                     21-09-01 18:43
>                     Please respond
>                     to Michael
>                     Schoberg
> 
> 
> 
> I can see the point of not involving a new ACK request for [DATA] & [R2T].
> However, it would be nice to have something like [CmdSN, ExpStatSN][StatSN,
> ExpCmdSN] for [DATA] & [R2T].  As it stands now, only when the [SCSI
> Response] is acknowledged for the request that originated the [R2T] &
> [Data]
> messages can you assume [R2T] & [Data] was successfully processed.  This
> means you have hold onto a lot of associations in that updating ExpStatSN
> will ACK more than just status messages.
> 
> The nice thing about the [CmdSN, ExpStatSN][StatSN, ExpCmdSN] mechanism was
> that it allowed for separate processing queues; status messages can be
> unaware of the command that originated them.  This disassociation allowed
> for a simpler implementation (aka hardware assist).
> 
> One possibility would be to split the [CmdSN, ExpStatSN][StatSN, ExpCmdSN]
> into 16 bit fields rather than 32.  Unless there's a genuine feeling that
> more than 65K requests could be outstanding within session, I don't see a
> problem.  The extra 32 bits freed up could be used for a Data/R2T_SN ACK
> and
> would allow a separate queue independent of [CmdSN, ExpStatSN][StatSN,
> ExpCmdSN].  Since each queue doesn't have to maintain state knowledge of
> the
> other, it allows for simpler design.
> 
> Just and idea.
> 
> : -----Original Message-----
> : From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
> : Sent: Friday, September 21, 2001 9:41 AM
> : To: ips@ece.cmu.edu
> : Subject: Re: iSCSI: SNACK R2T/Data
> :
> :
> :
> : Santosh,
> :
> : None of your arguments makes much sense.
> : The ACK (a third form of SNACK) can be done at explicit
> : boundaries and only
> : by initiators
> : supporting the within command class.  For most of the
> : initiator it will
> : require 0 additional resources as they either
> : don't do they recovery, or the input data is short (the ack
> : is not needed
> : at the end as the status ack acks the data too).
> : For the ones that have long exchanges (tape and 3rd party) it is a big
> : help.
> :
> : But you did not fail (again) to make your point.
> :
> : Julo
> :
> :
> :
> :
> :                     Santosh Rao
> :
> :                     <santoshr@cup.       To:
> : ips@ece.cmu.edu
> :                     hp.com>              cc:
> :
> :                     Sent by:             Subject:     Re:
> : iSCSI: SNACK R2T/Data
> :                     owner-ips@ece.
> :
> :                     cmu.edu
> :
> :
> :
> :
> :
> :                     20-09-01 18:54
> :
> :                     Please respond
> :
> :                     to Santosh Rao
> :
> :
> :
> :
> :
> :
> :
> :
> :
> : Matthew,
> :
> : We have gone through this thread of discussion regarding
> : DataSNa long time
> : back on
> : ips and the consensus has been that I/O transfer sizes  are not large
> : enough to
> : justify OUT_OF_BAND acknowledgement of data. [achieved by having the
> : initiator
> : generate NOP-OUTs in response to received data pdu's to send
> : a DataSN ack.]
> :
> : The primary dis-advantage with that scheme was that the data
> : ack's were
> : being
> : generated out-of-band, and therefore, implied extra cost in terms of
> : initiator
> : resources for each I/O, as well as increased wire traffic per I/O,
> : performance
> : penalty, etc.
> :
> : I am opposed to the draft requiring initiators to send
> : out-of-band ack's to
> : data
> : pdu's through the use of initiator generated NOP-OUT pdus. This is a
> : performance
> : penalty, a resource overhead, and not really justified in
> : most I/O traffic
> : due to
> : the avg. data xfer sizes.
> :
> : Regards,
> : Santosh
> :
> :
> : Julian Satran wrote:
> :
> : > Matthew,
> : >
> : > I am also in favor of such a mechanism.  It is easy to
> : implement (another
> : > form of SNACK) and we have already built-in a mechanism by which an
> : inbound
> : > stream can be marked for acking (the inbound sequence
> : separator F bit).
> : > It can also be specified as mandated only if the
> : within-command recovery
> : > class is present.
> : >
> : > However I am reluctant to open again this issue except if
> : there are more
> : > supporters. Data ACKs where hastily dropped at the interim
> : meeting in
> : > Orlando.  I recall Somesh Gupta, Pierre Labat and Santosh
> : Rao as being
> : very
> : > vocal against them (arguing complexity) and carrying the
> : room with them.
> : >
> : > Julo
> : >
> : >
> : >                     "BURBRIDGE,MATTH
> : >                     EW                     To:     Julian
> : Satran/Haifa/IBM@IBMIL,
> : >                     (HP-UnitedKingdo        ips@ece.cmu.edu
> : >                     m,ex2)"                cc:
> : >                     <matthew_burbrid       Subject:     RE:
> : iSCSI: SNACK
> : R2T/Data
> : >                     ge@hp.com>
> : >
> : >                     19-09-01 17:25
> : >                     Please respond
> : >                     to
> : >                     "BURBRIDGE,MATTH
> : >                     EW
> : >                     (HP-UnitedKingdo
> : >                     m,ex2)"
> : >
> : >
> : >
> : > I am very much in favour of having a positive ACK mechanism
> : to control
> : > buffer resources at the target.  If there is a very large
> : transfer (e.g.
> : 1
> : > Mb) then the sender can release buffer space once it knows that the
> : > receiver
> : > has received the data.  It is worth pointing out that this
> : mechanism is
> : for
> : > buffer control and is not for flow control which, as we all know, is
> : > handled
> : > by TCP.
> : >
> : > Cheers
> : >
> : > Matthew Burbridge
> : > Senior Development Engineer
> : > NIS-Bristol
> : > Hewlett Packard
> : > Telnet: 312 7010
> : > E-mail: matthewb@bri.hp.com
> : >
> : > -----Original Message-----
> : > From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
> : > Sent: Wednesday, September 19, 2001 6:28 AM
> : > To: ips@ece.cmu.edu
> : > Subject: Re: iSCSI: SNACK R2T/Data
> : >
> : > There is no ACK mechanism. The group wisdom was that there
> : is no need for
> : > one.
> : > Incoming data and R2Ts are not acked (a mechanism that did
> : that existed
> : and
> : > was based on NOP-Out).
> : >
> : > Julo
> : >
> : > Michael Schoberg <michael_schoberg@cnt.com> on 18-09-2001 19:09:51
> : >
> : > Please respond to Michael Schoberg <michael_schoberg@cnt.com>
> : >
> : > To:   "'ips@ece.cmu.edu'" <ips@ece.cmu.edu>, Julian
> : Satran/Haifa/IBM@IBMIL
> : > cc:
> : > Subject:  iSCSI: SNACK R2T/Data
> : >
> : > Old subject, but I couldn't find any discussion on this:
> : >
> : > When does the target know it no longer needs to hold R2T &
> : Data PDUs?
> : > StatSN responses are acknowledged through the ExpStatSN
> : field received in
> : > future I->T requests.  What's the acknowledgement method
> : for R2T & Data
> : > PDUs?  Is it tied to the original request and acknowledged
> : through the
> : > ExpStatSN acknowledgment of the request's response?
> : >
> : > Thanks.
> :
> :  - santoshr.vcf
> :
> :
> :