iSCSI:flow control, acknowledgement, and a deterministic recovery

["Douglas Otis" <dotis@sanlight.net>]

Ver 5 Pg. 10

   "Command numbering is session-wide and is used for ordered command
   delivery over multiple connections.  It can also be used as a
   mechanism for command flow control over a session."

"1.2.2.1 Command Numbering and Acknowledging

   iSCSI supports ordered command delivery within a session.  All
   commands (initiator-to-target) are numbered."

   "Commands in transit from the initiator SCSI layer to the SCSI target
   layer are numbered by iSCSI; the number is carried by the iSCSI PDU
   as CmdSN (Command-Sequence-Number).  The numbering is session-wide.
   All iSCSI PDUs that have a task association carry this number. CmdSNs
   are allocated by the initiator iSCSI within a 32-bit unsigned counter
   (modulo 2**32).  The value 0 is reserved and used to mean immediate
   delivery. Comparisons and arithmetic on CmdSN SHOULD use Serial
   Number Arithmetic as defined in [RFC1982] where SERIAL_BITS = 32.

   Not covered in this document are he means by which the SCSI layer may
   request immediate delivery for a command or by which iSCSI will
   decide by itself to mark a PDU for immediate delivery."


Not all commands are serialized as a result of serial number 0 representing
a special case for immediate delivery.  This has an impact on flow control,
acknowledgement, and a deterministic recovery in the face of an error
situation.  With the exception of those commands with null serialization,
all commands MUST be sequenced at the point of network aggregation described
here as a PDU sequencer that issues commands to the SCSI target.  This focal
point is likely to find situations where its normal operation is curtailed.

	1) Prolonged device operation resulting in a resource constraint.
	2) Digest Error causing a missing sequence.
	3) Connection loss causing a missing sequence.

The technique of using a null sequence to bypass the sequencer has some
inherent problems.  In first case, those queued commands MAY become invalid
following management that terminates prolonged operation with a command that
has bypassed the sequencer.  Those invalidated commands queued within the
sequencer can not be cancelled in an orderly manner within the existing
scheme.  This sequencer MUST be used as defined in the iSCSI proposal and,
as a result, these queued commands are beyond SCSI and iSCSI controls.
Issuing a null sequence command followed by a replicate serialized command
will have differing results depending on the target but will not result in a
deterministic treatment of these pending commands.

The use of the sequencer bypass technique (null serialization) should signal
an extreme measure where logically, those commands being bypassed become
suspect.  The conservative approach to this situation would be to reject all
bypassed commands.  As a result of this conservative behavior, a technique
that does not use a null sequence would be to institute a flag such as
"Exigent" that signals an extreme condition exists and that all pending
commands within the sequencer are to be rejected.

Note:
In addition to checking for the next PDU sequence, the sequencer should be
checking for PDUs with a serialization number that are prior to the desired
next sequence.  This examination would look something like this:

if ( (sequencer CmdSN - next sequence CmdSN ) > 2^(SERIAL_BITS - 1))
	{
	reject_pdu(CmdSN, SEQUENCER_INVALIDATION);
	}

if (sequencer CmdSN == next sequence CmdSN)
	{
	send_pdu(CmdSN);
	next sequence++;
	}

Upon receipt of a PDU flagged as "Exigent", the 'next sequence' value
immediately becomes the serial number of this command as well as ExpCmd
advancing to this value plus one.  The effect of this is to have the
sequencer reject all pending commands up to the "Exigent" command.  This has
the benefit of removing these now suspect commands as well as allowing this
highly urgent command to be sent to the target immediately without
accidentally affecting subsequent commands as is now possible.

As it is now, one other possible use of a null sequence would be to always
bypass the sequencer as perhaps the sequencer is viewed as unnecessary.  The
use of zero to allow commands to bypass the sequencer then represents a
problem with respect to resource management as now the flow control scheme
no longer works.  If bypassing the sequencer is the desired behavior and
this command will not impact validity of those commands serialized prior to
this command, then this PDU flagged "Casual" would allow this command to be
issued directly to the target.  These commands should still include a serial
number to allow flow control and acknowledgement to remain functional.  The
task of acknowledgement would be to comprise a min-max list of those
commands sent and to look for the highest sequential value.

In the case of a lost connection, waiting to time-out on those holes created
within the sequencer would be one method of handling this situation.  If
there was a means of rejecting those commands within the sequencer using an
"Exigent" command would mean there are no holes left to fill.  Those
commands received would be rejected and the initiator could then resend
these commands on a different connection without stumbling through a process
of repeated timeouts.  Should the method used to recover from a digestion
error mean terminating the connection, then those commands can also be
quickly shifted over to a new connection.

This does not resolve all issues created in a error event, but it does
provide a simpler solution for most of those events concerning the sequencer
and gets rid of the special case handing of the command serial number.  Not
having flow control, acknowledgement, and a method of dealing with queued
commands appear as a serious flaw in the present protocol.  I hope I have
presented this in a clear manner and I am interested in finding how others
deal with these situations.

Doug