Re: iSCSI:out-of-order notification proposal

["Mallikarjun C." <cbm@rose.hp.com>]

Sandeep,

Some comments on your proposal.

- The "strict_order" flag that you mention appears to carry
  information that's already contained by the SCSI task attributes
  (ATTR field).  

- This proposal requires the target iSCSI layer to look at more
  variables (SCSI-level information in that) for making its
  sequencing decisions. 

- Simple commands (intended to be) received prior to an Ordered 
  command must be executed before the Ordered one.  If you lose a 
  Simple command, you must wait for it before you act on the Ordered 
  that you received out-of-order.

This appears more an implementaion optimization where desired than
something we want to get into the draft.  You are proposing that 
even in the absence of errors, iSCSI should make ordering decisions
based on SCSI task attributes.  I disagree with that.
--
Mallikarjun 


Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668	Hewlett-Packard, Roseville.
cbm@rose.hp.com


>This is a proposal to allow the initiator to inform the target 
>if out-of-order execution within the command stream is possible.
>
>The target execution can rotate between "in-cmdSN-order" and 
>"out-of-order" during runtime as informed by the initiator..
>
>Appreciate comments on subtleties that I may have missed.
>
>thanks,
>-Sandeep
>
>http://ips.pdl.cs.cmu.edu/mail/msg03152.html
>   by Costa provides a good summary of the issue at hand.
>
>http://ips.pdl.cs.cmu.edu/mail/msg04255.html
>  David provides the "new" reqts.  In particular, this one
>
>   > MUST specify the ability to preserve ordered delivery
>   > of SCSI commands even in the presence of transport
>   > errors.  A mechanism MUST be provided to allow
>   > Initiators and Targets to negotiate this preservation
>   > on a per-session or finer granularity basis
>
>Note :
>======
>1) This does not rely on SCSI cmdRN, but operates at iSCSI level.
>2) Flow control using cmdSN works as designed.
>3) This solution is not a per-session negotiation option but can be
>   disabled and re-enabled again at "runtime" by the initiator if it
>   notices that Ordered/HOQ tasks (or any other need) have entered
>   the iSCSI command stream which is being dispatched.
>
>Problem:
>=========
>In case of out-of-order arrival or digest errors, it is NOT possible
>to know if the initiator had sent an ordered command before the one
>which was received.
>
>Solution:
>=========
>To notify target of presence of in-flight ordered commands, we set
>a flag on *every* PDU following the ordered command *until* the target
>moves it expCmdSN above the cmdSN of the ordered command.  The
>expCmdSN indicates target has found the ordered command.
>
>( Those familiar with "ECN over IP" (Floyd, et al) may see this is
>similar to how a congestion bit keeps being set until the sender acks
>that it has received the notification)
>
>Figuratively:
>=============
>So, assume there was a new "strict_order" flag in the BHS.
>In the figure below, braces shows value of (cmdSN, strict_order)
>
>Initiator                       Target
>---------                       -------
>
>simple cmd (cmdSN=100, strict_order=0) ->
>simple cmd (101, 0) ->
>simple cmd (102, 0) ->
>
>   these may get reordered or have digest errors ->
>
>                        target executes as they arrive
>                             exec simple cmd(102, 0)
>                             exec simple cmd(100, 0)
>                             exec simple cmd(101, 0)
>
>Now say the initiator wants to send a HOQ task.
>It sets strict_order=1 on all PDUs
>
>ordered cmd (103, 1) ->
>simple  cmd (104, 1) ->
>simple  cmd (105, 1) ->
>                        in case of reordering or digest errors
>                        target must wait & execute in cmdSN order!
>simple  cmd (106, 1) ->
>simple  cmd (107, 1) ->
>
>         <---- now target sends expCmdSN=103
>
>This implies target has seen command(cmdSN=103) and target will do the
>appropriate ordering and delivery to SCSI layer.  This is left to
>the target implementation to tackle.
>
>Initiator checks (expCmdSN >= cmdSN of ordered cmd) and then resets
>strict_order to zero for all subsequent PDUs.
>
>simple  cmd (108, 0) ->
>simple  cmd (109, 0) ->
>
>Other issues:
>=============
>If the basic scheme is ok, then we could later tackle other questions
>"what about multiple ordered commands" and the like..