RE: iSCSI: NOPs should not have a CmdSN

[Black_David@emc.com]

>    If immediate delivery is used with task management commands, these
>    commands may reach the SCSI target task management before the tasks
they
>    are supposed to act upon.  However, their CmdSN is a good reference for
>    what commands the immediate command was supposed to act upon; to
achieve
>    this function the task management command MUST be the same CmdSN as
that
>    which would have been given to the next non-immediate command.

I hope this is only a wording issue, but I'm going to supply a long
explanation
just in case.  I'm concerned that "to achieve this function" may be
read as modifying the subsequent MUST, especially in combination with the
sentence prior to it.

To help explain the above text, it would be good to start by adding a few
sentences describing what SAM-2 requires for behavior of task management
commands:
- SAM-2 assumes that all commands are delivered to the Target in the order
	that they were issued by the Initiator.
- Therefore all task management commands MUST exhibit behavior at the
	Initiator SCSI interface to iSCSI that is consistent with this
ordered
	delivery.
For example, if a command C is issued, followed by an Abort Task of C, two
mis-ordering scenarios are possible:

- The Abort Task reaches the Target before Command C.
- The Abort Task reaches the Target after Command C, but the
	Abort Task response reaches the Initiator before the response
	to Command C.

Both cases could result in the successful completion of Abort Task (no
task to abort) being delivered to SCSI by the iSCSI Initiator followed by
delivery of the response to Command C.  This MUST NOT happen.  Most
of the MUSTs and MUST NOTs required to prevent this belong in the
specification of task management, but some wording cleanups would
help here:

Old:
>    However, their CmdSN is a good reference for
>    what commands the immediate command was supposed to act upon; 
New:
  To deal with this, the CmdSN in a task management
  command serves as a marker to indicated the command's
  position in the sequence of issued commands.

Old:
>  to achieve
>    this function the task management command MUST be the same CmdSN
>    as that which would have been given to the next non-immediate command.
New:
  The CmdSN for a task management command issued for immediate
  delivery MUST be the CmdSN that would be assigned to the next
  non-immediate command.

Also, I have a serious concern about the following text:

>   Please note that the number of commands used for immediate delivery is
>    not limited and their delivery to execution is not acknowledged through
>    the numbering scheme.  Immediate commands can be rejected by the iSCSI
>    target due to lack of resources. An iSCSI target MUST be able to handle
>    at least one immediate task management command and one immediate
>    non-task-management iSCSI request at any time.

As written, that looks like a source of interoperability problems because it
seems to allow a Target to silently ignore immediate commands in excess of
one
immediate task-management plus one immediate non-task management
command.  One of several things has to happen:
(1) Targets MUST be able to handle any and all immediate commands fired
	at them over and above the resources devoted to non-immediate
commands.
(2) There is some static limit (e.g., the one plus one above) to the number
of
	immediate commands that a target MUST handle.
(3) Same as (2), but the limit is negotiated.

(1) has nasty resource management consequences for the Target, but
I think that's what the current text actually requires :-(.

(2) looks like the easiest way out.  The MUST for the target should
be accompanied by a corresponding "MUST not issue concurrent
immediate commands beyond one task-management and one
non-task management command" plus some additional words
about how the Initiator can determine when a
subsequent immediate command would not be concurrent
with previously issued ones.

The simplest approach is to require responses to all immediate
commands, including NOP - a more clever approach might be
to take Matt's NOP case for updating CmdSN-related state
and explain how the Initiator could determine that the NOP
has been executed without requiring a response.  In order to
solve Matt's original problem, we would need to 
make that "one immediate task management
command plus one non-immediate task management command"
requirement apply *per connection*, not *per session* or
*per target*.

Thanks,
--David

> -----Original Message-----
> From:	Julian Satran [SMTP:Julian_Satran@il.ibm.com]
> Sent:	Saturday, July 14, 2001 11:10 AM
> To:	ips@ece.cmu.edu
> Subject:	RE: iSCSI: NOPs should not have a CmdSN
> 
> 
> The need to reject immediate commands came up several times during
> recovery
> discussions.
> NOP with CmdsN was there to allow checking the "delivery machine"
> end-to-end.
> The text for CmdSN in case of retry had also some points that needed
> clarification.
> Here is the (more liberal)  text for numbering  commands:
> 
> 
> 1.1.1.1   Command Numbering and Acknowledging
> 
>    iSCSI supports ordered command delivery within a session.  All commands
>    (initiator-to-target PDUs) are numbered.
> 
>    Any SCSI activity is related to a task (SAM-2). The task is identified
>    by the Initiator Task Tag for the life of the task.
> 
>    Commands in transit from the initiator to the target layer are numbered
>    by iSCSI; the number is carried by the iSCSI PDU as CmdSN
>    (Command-Sequence-Number).  The numbering is session-wide.  All
> outgoing
>    iSCSI PDUs that have a task association, except Data-Out, carry this
>    number. CmdSNs are allocated by the initiator iSCSI within a 32-bit
>    unsigned counter (modulo 2**32). Comparisons and arithmetic on CmdSN
>    SHOULD use Serial Number Arithmetic as defined in [RFC1982] where
>    SERIAL_BITS = 32.
> 
>    Commands meant for immediate delivery are marked as such through an
>    immediate delivery flag. They MAY carry any CmdSN. The CmdSN is not
>    advanced for commands marked for immediate delivery.
> 
>    If immediate delivery is used with task management commands, these
>    commands may reach the SCSI target task management before the tasks
> they
>    are supposed to act upon.  However, their CmdSN is a good reference for
>    what commands the immediate command was supposed to act upon; to
> achieve
>    this function the task management command MUST be the same CmdSN as
> that
>    which would have been given to the next non-immediate command.
> 
>    Not covered in this document are the means by which one may request
>    immediate delivery for a command or by which iSCSI will decide by
> itself
>    to mark a PDU for immediate delivery.
> 
>    Please note that the number of commands used for immediate delivery is
>    not limited and their delivery to execution is not acknowledged through
>    the numbering scheme.  Immediate commands can be rejected by the iSCSI
>    target due to lack of resources. An iSCSI target MUST be able to handle
>    at least one immediate task management command and one immediate
>    non-task-management iSCSI request at any time.
> 
>    Except for the commands marked for immediate delivery the iSCSI target
>    layer MUST deliver the commands for execution in the order specified by
>    CmdSN. Commands marked for immediate delivery may be handed over by the
>    iSCSI target layer for execution as soon as detected. iSCSI may avoid
>    delivering some command for execution if so required by some prior SCSI
>    or iSCSI action (e.g., clear task set Task Management request received
>    before all the commands it was supposed to act on).
> 
>    The initiator and target are assumed to have three registers, unique
>    session wide, that define the numbering mechanism:
> 
>        - CmdSN - the current command Sequence Number advanced by 1 on each
>       command shipped except for commands marked for immediate delivery.
>        - ExpCmdSN - the next expected command by the target. The target
>       acknowledges all commands up to but not including this number. The
>       target iSCSI layer sets the ExpCmdSN to the largest non-immediate
>       CmdSN that it is able to deliver for execution plus 1 (no holes in
>       the CmdSN sequence).
>        - MaxCmdSN - the maximum number to be shipped. The queuing capacity
>       of the receiving iSCSI layer is MaxCmdSN - ExpCmdSN + 1.
> 
>    ExpCmdSN and MaxCmdSN are derived from target-to-initiator PDU fields.
> 
>    MaxCmdSN and ExpCmdSN fields are processed as follows:
> 
>       -if the PDU MaxCmdSN is less than the PDU ExpCmdSN-1 (in Serial
>       Arithmetic Sense and with a difference bounded by 2**31-1), they are
>       both ignored
>       -if the PDU MaxCmdSN is less than the local MaxCmdSN (in Serial
>       Arithmetic Sense and with a difference bounded by 2**31-1), it is
>       ignored; else it updates the local MaxCmdSN
>       -if the PDU ExpCmdSN is less than the local ExpCmdSN (in Serial
>       Arithmetic Sense and with a difference bounded by 2**31-1), it is
>       ignored; else it updates the local ExpCmdSN
> 
>    This sequence is required as updates may arrive out of order because
>    they travel on different TCP connections.
> 
>    The target MUST NOT transmit a MaxCmdSN that is more than 2**31 - 1
>    above the last ExpCmdSN.  For non-immediate commands, the CmdSN field
>    can take any value from ExpCmdSN to MaxCmdSN. The target MUST silently
>    ignore any non-immediate command outside this range or non-immediate
>    duplicates within the range that have not been flagged with the retry
>    bit (the X bit in the opcode).
> 
>    iSCSI initiators and targets MUST support the command numbering scheme.
> 
>    A numbered iSCSI command will not change its allocated CmdSN regardless
>    of the number of times and circumstances in which it is reissued.  At
>    the target, it is assumed that CmdSN is relevant only while the command
>    has not created any execution state (can't find the Initiator Task
> Tag).
>    Afterwards CmdSN becomes irrelevant.  Testing for execution state is
>    assumed to precede any other action at the target and is followed by
>    ordering and delivery if no execution state is found or delivery if
>    execution state is found. Immediate commands can't be retried unless
>    there is execution state available at the target for them (they are
>    rejected for retry if the target can't find the Initiator Task Tag).
> 
> 
>    Julo