Re: iSCSI: CmdSN and Retry

[Santosh Rao <santoshr@cup.hp.com>]

> I certainly understand the need of doing data/status recovery and the
> argument of "local" recovery being simpler.  However, this comes with 
> heavy cost of performance when pipelined design demands 100,000 IOs per 
> second on a network with long delay.  For services and responses 
> happening a few times per second, it is OK to hold on the resources 
> until we are certain the ACK is returned.  However, in the example 
> above, after completing command 1 if a target can't start command 2 
> until the status for 1 is ACK'ed, the wait can
> be 100 milliseconds on a network with long delay.  The wait make it
> impossible to have large number of IOs in the pipeline. By mandating
> data/status recovery in iSCSI, we change the pipelined command 
> execution to interlock handshakes.  
> As I have said in the previous email, an initiator
> will never send an command which depends on the success of a previous
> command.  This fact makes the pipeline execution in a target possible.

YP,

I agree with the above and have been saying the same thing in a seperate
thread. (See : "Command Ordering Proposal"
http://ips.pdl.cs.cmu.edu/mail/msg03171.html)

(The corner cases stated simply point out issues with the SNs as they exist
in the draft today.) The bigger question of why iSCSI is attempting to
provide ordering when its peers do not is still un-answered.

iSCSI is trying to enforce strict ordering which is not required due to the
following reasons :

1) Other SCSI Transport Protocols (FC, pSCSI) do not guarantee strict
ordering across commands. 

2) Most (All ?) operating system SCSI Stacks enforce ordering above the
SCSI ULP and do not depend on the SCSI Stack to provide end-to-end
ordering across commands. (The exception to this is ordering of commands
followed by their aborts).

3) In the absence of (1) & (2) above, applications cannot depend on SCSI
transport protocols to provide strict end-to-end ordering across commands,
since they are transport agnostic and do not wish to base ordering
assumptions on the transport they operate on.  (i.e. appln code enforces
ordering if it operates on FC and pSCSI and uses iSCSI ordering when it
runs above iSCSI !)

4) Error recovery scenarios needed to maintain ordering across transport
errors, resource allocation errors, errors at the target, digest errors
and format errors becomes extremely complex, stalls all pipelined
operations and in some cases cannot resolve the out-of-order situation
induced unless the target executes 1 command at a time and ONLY starts on
the next command after a StatSN ACK for the previous one.

5) Strict ordering results in performance penalties such as :
a) non-concurrent command execution at the targets.
b) stalled TCP connections in a session when a connection turns faulty.
c) all commands need to be stalled and re-initiated on any I/O error in an
attempt to maintain ordering.
d) Flow Control mechanisms like QUEUE FULL error recovery will change to
complete oscillating algorithms with the initiator completely stopping all
further I/Os until order is restored in the sequence they were originally
sent.

6) Strict ordering is not required in the majority if not all cases, since
most (all ?) O.S' do not enforce strict ordering or do anything 
special in their error recovery to  maintain strict ordering.

7) Several targets treat simple and ordered tags in a similar manner
without differentiating and providing strict ordering for ordered tag
commands.

8) Majority of the I/O traffic (if not all) is simple tag commands and all
this ordering and error recovery for ordering will impact performance when
ordering was never required in the first place.

9) The use of ordering within a SCSI stack is not prevalent. IN those few
places where it is known to be in use, this is a static characteristic of
that SCSI stack, and such stacks can use single-connection sessions to
achive their ordering.

IOW, the majority of stacks do NOT require strict ordering. Hence, iSCSI
should optimize for this most common case and NOT penalize it. 

> 
> On a separate note, I really respect Santosh's fine-tooth analysis of the
> iSCSI draft.  But, in his arguments the fact that SCSI has been functional
> for the last 20 years was badly ignored.  

YP, the corner cases simply bring out problems in the draft as they exist
today. I have been consistently raising the bigger question about why
iSCSI is attempting to differ from its peers in issues like ordering,
overlapped data xfer's, etc. SCSI has been functioning over the last 20
years without applns depending on SCSI stacks for strict ordering or
deploying un-used features like overlapped data xfers. 

Applns will continue NOT to depend on scsi transports for the above since
they are written to be transport agnostic, and are based on lowest common
denominator support. (i.e. assume no ordering from O.S SCSI Stack).

> However, using StatSN to
> mandate data/status retry pays a great performance price. 

Agreed. 

> Both overlapped
> and out-of-order data transfers are allowed in SCSI (Check out the Modify
> Data Pointer extended message).  SCSI works fine without mandating
> non-overlapping transfers or data/status recovery.  

The Modify Data Pointers needs to be explicitly enabled through Mode
Select and I can't recollect seeing any instances of its usage.
As for FCP, overlapped data xfer's are not allowed. period.

Regards,
Santosh

-- 
#################################
Santosh Rao
Software Design Engineer,
HP, Cupertino.
email : santoshr@cup.hp.com
Phone : 408-447-3751
#################################