RE: iSCSI - positive data ack - change proposal

["Rod Harrison" <rod.harrison@windriver.com>]

John,

	I don't actually think the NOP approach is something one would want
to rely on, it is more something that can be done to help a target
that is about to stop because of resource limitations decide on some
things to free.

	That said I don't agree that even in the tape world long data reads
will be the norm. It is usual for host operating systems to break long
transfers in to multiple smaller transfers. Often because of
limitations in HBA DMA etc. Typically HBAs indicate their maximum
transfer capability and the host SCSI layer accommodates them.

	If we believe we really need to cater for long data reads, and it is
beginning to sound like we do, then one of the in-band ACK proposals
is the way to go.

	- Rod

-----Original Message-----
From: John Hufferd [mailto:hufferd@us.ibm.com]
Sent: Tuesday, September 25, 2001 7:26 PM
To: Rod Harrison
Cc: ips@ece.cmu.edu
Subject: RE: iSCSI - positive data ack - change proposal



Rod,
Do you believe that your approach would be approprate when dealing
with
Tape?  That is an environment where the probability of long Data Reads
is
significantly high.  Also it is an environment where the higher level
recovery is non-existent in many/most environments.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: hufferd@us.ibm.com


"Rod Harrison" <rod.harrison@windriver.com>@ece.cmu.edu on 09/25/2001
03:10:37 AM

Sent by:  owner-ips@ece.cmu.edu


To:   "Santosh Rao" <santoshr@cup.hp.com>, <ips@ece.cmu.edu>
cc:
Subject:  RE: iSCSI - positive data ack - change proposal




     I agree, I don't believe we need to positively ACK DATA-IN PDUs.
IIRC
the WG consensus has been that a target can free its resources
whenever it chooses, reconstructing data from the media if necessary.
I also agree that long DATA-IN sequences are unlikely enough to make a
positive ACK unnecessary. However, consider the following.

     In the unlikely event of a DATA-IN sequence being long enough to
push
the target towards some critical resource limitation, the target could
get some measure of confidence that it can release resources by
sending a non-immediate NOP-IN ping request on the same connection. It
is guaranteed that the initiator won't process the NOP-IN until all
the DATA-IN PDUs have been read from the wire and processed. The
initiator will send a NOP-OUT ping response on the same connection to
the target. The initiator isn't going to do anything with the data
except DMA to the host, so when the target receives the ping response
it knows that worst case a couple of DMAs might be outstanding. If the
target now frees the resources it had associated with some, or all, of
the DATA-IN PDUs sent before the NOP-IN I believe the chance of them
being needed again is very small. Worst case is that the target
receives a data SNACK after freeing some of the resources, in which
case the target reconstructs the data from the media, or simply sends
a SNACK rejected ASC read error and forces the initiator into a higher
level of recovery. I think this is unlikely enough to be acceptable.

     - Rod



-----Original Message-----
From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
Santosh Rao
Sent: Tuesday, September 25, 2001 2:16 AM
To: ips@ece.cmu.edu
Subject: Re: iSCSI - positive data ack - change proposal


Julian,

I'm opposed to any use of out-of-band techniques by initiators to
perform acks of data-in pdu's.

If at all this level of data ack is required, I would like to suggest
a simpler in-band ack mechanism which is that targets number all PDUs
that are transmitted from target to initiator with a sequence number.
(say, InboundSN) on a per connection basis. Initiators can then
acknowledge inbound PDUs as they are received by sending ExpInboundSN
updates on outbound PDUs.

Such a technique is simpler to implement for initiators, less
expensive in terms of resources and does not impact performance.

Specific to your proposal below, I am concerned about the data ack's
being based at sequence boundaries (thru the use of F bit in data-in
PDU to indicate "Data ACK" required). This may result in data ack's
being generated too frequently. The data ack boundaries should be
signalled through a seperate bit rather than over-riding the semantics
of the F bit.

That said, I am opposed to the below proposal for all the reasons
outlined earlier + the above. IF a data ack scheme is required (I
did'nt hear enough "yes" calls to over-ride the Orlando WG consensus
(?) ), then, I would favor an in-band technique such as the one
suggested in this mail, wherein no extra PDUs are required from
initiator to target for the sole purpose of acknowledging another
inbound PDU or set of inbound PDUs.

Thanks,
Santosh


Julian Satran wrote:

> Here is updated version (in the previous I had excluded the
sequences ended
> with Status with no good reason.
>
> Julo
>
> (See attached file: ack.txt)
>
> ----- Forwarded by Julian Satran/Haifa/IBM on 23-09-01 04:41 -----
>
>
>                     Julian Satran
>                                          To:     ips@ece.cmu.edu
>                     22-09-2001           cc:
>                     14:04                From:   Julian
Satran/Haifa/IBM@IBMIL
>                                          Subject:     iSCSI -
positive data ack - change
>                                           proposal
>
>
>
>
> Dear colleagues,
>
> As  I mentioned earlier all the elements needed for positive
data-ack are
> already in place.
>
> I am suggesting the following changes to the document to reintroduce
the
> data-ACK.
>
> Comments?
>
> Julo
>
> **** Attachment ack.txt has been removed from this note on 23
September
> 2001 by Julian Satran ****
>
>   ------------------------------------------------------------------
------
> Within 2.2.2.3
> ---------------
>
> Initiators MAY also acknowledge received data and reduce by this the
resources a target needs to dedicate to data recovery if target and
initiator support this type of recovery.
>
> ------
> 3.7.1 F (Final) Bit
>
> For outgoing data, this bit is 1 for the last PDU of unsolicited
data or the last PDU of a sequence answering an R2T.
>
> For incoming data, this bit is 1 for the last input (read) data PDU
of a sequence.  Input can be split in several sequences each one
having it's own F bit. Splitting in sequences does not affect DataSN
counting on Data-In PDUs but MAY be used as a "change direction"
indication for Bidirectional operations that need such a change and/or
end of recoverable sequences by targets with a limited retransmission
buffer.  Upon receiving an Data-In PDU with the F set to 1 in a
session with ErrorRecoveryLevel 1 or higher the initiator MUST issue a
DataACK type of SNACK indicating the next expected DataSN for this
task.
>
> For Bidirectional operations, the F bit is 1 both for the end of the
input sequences as well as the end of the output sequences.
>
>
> -----------
>
> 3.16  SNACK Request
>
> Byte /    0       |       1       |       2       |       3       |
>    /              |               |               |               |
>   |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|
>   +---------------+---------------+---------------+---------------+
>  0|0|1| 0x10      |1|Rsrvd| Type  | Reserved                      |
>   +---------------+---------------+---------------+---------------+
>  4/ Reserved                                                      /
>  +/                                                               /
>   +---------------+---------------+---------------+---------------+
> 16| Initiator Task Tag or 0xffffffff                              |
>   +---------------+---------------+---------------+---------------+
> 20| BegRun                                                        |
>   +---------------+---------------+---------------+---------------+
> 24| RunLength                                                     |
>   +---------------+---------------+---------------+---------------+
> 28| ExpStatSN                                                     |
>   +---------------+---------------+---------------+---------------+
> 32| Reserved                                                      |
>   +---------------+---------------+---------------+---------------+
> 36| ExpDataSN or Reserved                                         |
>   +---------------+---------------+---------------+---------------+
> 32/ Reserved                                                      /
>  +/                                                               /
>   +---------------+---------------+---------------+---------------+
> 48
>
> Support for SNACK is optional.
>
> SNACK request is used to request retransmission of
numbered-responses, data or R2T PDUs from the target.  The SNACK
request indicates to the target the missed numbered-response or data
run, where the run is composed of an initial missed StatSN, DataSN or
R2TSN and the number of additional missed Status, Data or R2T PDUs (0
means only the initial).
>
> The numbered-response, Data or R2T PDUs requested by a SNACK have to
be delivered as exact replicas of the ones the initiator missed
including all its flags.
>
> Any SNACK requesting a numbered-response, Data or R2T that was not
sent by the target MUST be rejected with a reason code of "Invalid
SNACK".
>
> SNACK is also used to positively acknowledge Data-In PDUs.
>
> 3.16.1 Type
>
> This field encodes the SNACK function as follows:
>
> 0-Data/R2T SNACK - requesting retransmission of a Data-In or R2T PDU
> 1-Status SNACK - requesting retransmission of a numbered response
> 2-DataACK - positively acknowledges Data-In PDUs
>
> All other values are reserved.
>
> Data/R2T SNACK for a command MUST precede status acknowledgement for
the given command.
>
> For a Data/R2T SNACK or a DataACK, the Initiator Task Tag MUST be
set to the Initiator Task Tag of the referenced Command. Otherwise, it
is reserved.
>
> For a Status SNACK the ExpDataSN field is reserved.
>
> An iSCSI target that does not support recovery within connection MAY
discard status SNACK. If the target supports command recovery within
session it MAY discard the SNACK after which it MUST issue an
Asynchronous Message PDU with an iSCSI event indicating "Request
Logout".
>
> If an initiator operates at ErrorRecoveryLevel 1 or higher it MUST
issue a SNACK of type DataACK after receiving a Data-In PDU with the F
bit set to 1.
>
> 3.16.2 BegRun
>
> First missed DataSN, R2TSN or StatSN
>
> 3.16.3 RunLength
>
> RunLength is the number of sequential missed DataSN, R2TSN or
StatSN. RunLength 0 signals that all Data-In, R2T or Response PDUs
carrying numbers equal or greater to BegRun have to be resent.