RE: iSCSI:flow control, acknowledgement, and a deterministic recovery

[Black_David@emc.com]

> Ver 5 Page 11:
> 
>    "The iSCSI target layer MUST deliver the commands to the SCSI target
>    layer in the order specified by CmdSN."
> 
> I am not making an assumption about there being a CmdSN sequence function.
> It is a MUST within the proposal!  Yes, you can bridge holes within CmdSN
> sequence using pointers and that would be expected and I described this as
a
> list.  This list still must be sorted between sequential and immediate
> especially when multiple connections are used to determine a next in
> sequence.

Aha ... here's a problem.  The text Doug quotes can't apply to existing
immediate
commands, because the current use of a zero CmdSN would require them to be
delivered before all other commands - what if the first immediate command
shows
up after 3000 non-immediate commands?  If we adopt the proposal to use
regular
CmdSNs for immediate commands, we'll need some text indicating that the
above MUST does NOT apply to immediate commands - i.e., iSCSI is expected
to deliver them to SCSI immediately after TCP delivers them to iSCSI no
matter
what sate the CmdSN window is in.  There is no need for rejection -
if worst comes to worst, a dummy command whose "deliver to SCSI"
operation is "do nothing" will keep Doug's sequencer happy even though
the actual command was delivered out of order.

> This list still must be sorted between sequential and immediate
> especially when multiple connections are used to determine a next in
> sequence.

That's one option.  Another is to have two lists so the immediate commands
don't get mixed with those for ordered delivery.  As I said, this is within
the realm of implementation choices.

> You would not want the creation of these lists to be blocking
> until the next in sequence is seen as you seem to imply.

I suspect Doug has misread what I wrote - most of the time, there's
no need to block for "next in sequence". OTOH, there is a "no free lunch"
principle in here that says Target resources are limited - if the Target is
temporarily out of resources, not much is going to happen until it gets
resources, and in that case, blocking is a reasonable first step.
Rejecting perfectly valid commands just to free resources doesn't seem
like the first thing that a Target should do (e.g., vs. waiting for SCSI to
finish something).

> You view this as a pointer list, again I agree.  The use of rejection
still greatly
> simplifies this process.

I still believe that "greatly simplifies" is implementation-specific until
someone
else speaks up in support of it.

> You are wrong about the use of CmdSN being limited to just
> the flow control window.

That's incorrect.  Once the command is ready for delivery to SCSI,
CmdSN is not needed to track it if there's some other way to do so.
"Deliver in CmdSN order" doesn't require that an implementation use
CmdSN to track that order, and the fact that immediate commands
put holes in this ordering may lead an implementation to use
something else.

> To not support rejects, you are then relying on the target to handle these
> now out of sequence commands.  

Immediate commands are "out of sequence" by definition - immediate
means deliver immediately without regard to CmdSN order.  Only the
target can do this, so I don't see any problem with relying on the Target
to do so.

>  In addition, that there be spare overhead allotted to handle these
> undetermined number of "immediate" commands.  

That's open to debate and depends on whether we require the CmdSN
to be within the window (as originally proposed) or allow one (more
than one?) beyond the window to be used (as Julian has proposed).

> You will then also not see any immediate acknowledgement of these
> "immediate" commands sent to the target unlike ALL other such commands.

In most of the cases, the TCP ACK will indicate that the bytes got there.
If there's an iSCSI CRC failure, then we're back to the CmdSN sequencing
to catch it and clean up, and that might not be "immediate" - but is that
going
to happen often enough to be worth any additional effort (rejects, immediate
ACKs, something else?) to optimize for performance?  I'm sceptical,
especially if a Cmd SACK/SNACK is proposed to correct this.

> Can you answer how many "immediate" commands are allowed?

Initiator determines this by appropriate management of usage of the CmdSN
window.
If it keeps N slots open in the window, N immediate commands can be sent
immediately.

> When would you expect these "immediate" commands to be acknowledged to
> allow more such commands?

When the resources from executing an immediate command are no longer
needed on the Target, MaxCmdSN would advance to allow an additional command
in, even if ExpCmdSN has not moved.  The Initiator has to spend some of its
time dealing with keeping ExpCmdSN moving (i.e., resend the command
at ExpCmdSN) because Targets are unlikely to support arbitrarily large
windows.

--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
black_david@emc.com       Mobile: +1 (978) 394-7754
---------------------------------------------------