iSCSI:out-of-order notification proposal

[Sandeep Joshi <sandeepj@research.bell-labs.com>]

This is a proposal to allow the initiator to inform the target 
if out-of-order execution within the command stream is possible.

The target execution can rotate between "in-cmdSN-order" and 
"out-of-order" during runtime as informed by the initiator..

Appreciate comments on subtleties that I may have missed.

thanks,
-Sandeep

http://ips.pdl.cs.cmu.edu/mail/msg03152.html
   by Costa provides a good summary of the issue at hand.

http://ips.pdl.cs.cmu.edu/mail/msg04255.html
  David provides the "new" reqts.  In particular, this one

   > MUST specify the ability to preserve ordered delivery
   > of SCSI commands even in the presence of transport
   > errors.  A mechanism MUST be provided to allow
   > Initiators and Targets to negotiate this preservation
   > on a per-session or finer granularity basis

Note :
======
1) This does not rely on SCSI cmdRN, but operates at iSCSI level.
2) Flow control using cmdSN works as designed.
3) This solution is not a per-session negotiation option but can be
   disabled and re-enabled again at "runtime" by the initiator if it
   notices that Ordered/HOQ tasks (or any other need) have entered
   the iSCSI command stream which is being dispatched.

Problem:
=========
In case of out-of-order arrival or digest errors, it is NOT possible
to know if the initiator had sent an ordered command before the one
which was received.

Solution:
=========
To notify target of presence of in-flight ordered commands, we set
a flag on *every* PDU following the ordered command *until* the target
moves it expCmdSN above the cmdSN of the ordered command.  The
expCmdSN indicates target has found the ordered command.

( Those familiar with "ECN over IP" (Floyd, et al) may see this is
similar to how a congestion bit keeps being set until the sender acks
that it has received the notification)

Figuratively:
=============
So, assume there was a new "strict_order" flag in the BHS.
In the figure below, braces shows value of (cmdSN, strict_order)

Initiator                       Target
---------                       -------

simple cmd (cmdSN=100, strict_order=0) ->
simple cmd (101, 0) ->
simple cmd (102, 0) ->

   these may get reordered or have digest errors ->

                        target executes as they arrive
                             exec simple cmd(102, 0)
                             exec simple cmd(100, 0)
                             exec simple cmd(101, 0)

Now say the initiator wants to send a HOQ task.
It sets strict_order=1 on all PDUs

ordered cmd (103, 1) ->
simple  cmd (104, 1) ->
simple  cmd (105, 1) ->
                        in case of reordering or digest errors
                        target must wait & execute in cmdSN order!
simple  cmd (106, 1) ->
simple  cmd (107, 1) ->

         <---- now target sends expCmdSN=103

This implies target has seen command(cmdSN=103) and target will do the
appropriate ordering and delivery to SCSI layer.  This is left to
the target implementation to tackle.

Initiator checks (expCmdSN >= cmdSN of ordered cmd) and then resets
strict_order to zero for all subsequent PDUs.

simple  cmd (108, 0) ->
simple  cmd (109, 0) ->

Other issues:
=============
If the basic scheme is ok, then we could later tackle other questions
"what about multiple ordered commands" and the like..