iSCSI: update on OOO CmdSNs/connection

["Mallikarjun C." <cbm@rose.hp.com>]

To those interested in this discussion:

Julian and I had a phone conversation on the topic
of OOO CmdSNs on a connection.  An update follows.

Julian agrees that there are valid error recovery
scenarios where CmdSNs will legitimately end up OOO
on a given connection.

OTOH, I agree with two of Julian's scenarios that he 
pointed out right away - the "cleaning command" (command
required to be sent after a retry copy to ensure flushing 
within 2^31 -1), and an immediate Logout posted with 
unacknowledged commands.  Neither of this can be shipped 
OOO - since the former undoes the flushing intent, and 
the latter breaks the rule that nothing more follows a 
Logout on the connection (and troublesome in other ways,
see below).  

In general, I share the concern with Julian that we 
have not closely scrutinized all possibilities.

With that said, something along the following lines
seemed reasonable -

A)Initiator MUST send commands in increasing order of 
  CmdSN on a connection if both the following are true -
	- operational ErrorRecoveryLevel is 0,
	- MaxConnections is negotiated to 1.
B)In all the other cases, initiator SHOULD send commands
  in increasing order of CmdSN on a connection.  It is 
  strongly encouraged that commands with out-of-order
  CmdSNs be sent on a connection only if they are 
  retransmitted commands due to digest error recovery 
  and connection recovery.

I also suggest the following upon further reflection-

C)Add wording in section 2.2.2.1 to mandate that
  the cleaning command MUST be sent in-order after 
  the retried command.
D)Warn clearly that sending an immediate Logout command 
  in the presence of other unacknowledged commands MAY 
  create inadvertent discarding of certain commands (even
  if it is a recovery Logout), and MAY cause protocol 
  errors leading to ungraceful shutdown of the connection.

Hopefully A will bring the determinism that Somesh was 
looking for certain design points.  B describes the more 
general n-connection session case.  C & D are fixes for 
two identfied areas (so far) which will break. 
 
Comments?
-- 
Mallikarjun 


Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668	Hewlett-Packard, Roseville.
cbm@rose.hp.com