RE: iSCSI:flow control, acknowledgement, and a deterministic recovery

[Black_David@emc.com]

Doug,

There are a couple of confusions in here:

TASK_SET_FULL *is* a flow control mechanism, albeit a
rather primitive one - the Initiator issues commands until
it gets a FULL response and hence knows the capacity
of the Target's queue.  The statement that the Target loses
resource control if it only uses TASK_SET_FULL is wrong,
although it is relying on code at or above the SCSI driver
on the Initiator to handle the flow control.

Santosh did not propose to remove CmdSN, although he
did point out an issue that needs attention - we need
to say something about the interaction of CmdSN with
TASK_SET_FULL since both mechanisms may be implemented.
In particular, it's possible to use CmdSN for delivery
ACKs and TASK_SET_FULL for flow control, although that's
a "peculiar" combination (and perhaps a SHOULD NOT?).

Beyond this, your entire second paragraph is well off
into the weeds.  Please be careful about taking the
discussion off onto wild tangents.

Thanks,
--David
---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
black_david@emc.com       Mobile: +1 (978) 394-7754
---------------------------------------------------

From:	Douglas Otis [dotis@sanlight.net]
Sent:	Tuesday, April 10, 2001 2:09 AM
To:	santoshr@cup.hp.com
Cc:	Ips
Subject:	RE: iSCSI:flow control, acknowledgement, and a deterministic
recovery

Santosh,

In a change recommending a flag and a valid CmdSN and not a null value, I
included a flag named "Casual" to provide this mode of operation you
describe which still allows flow control.  I would not wish to permit this
Casual mode, but as David feels such lists or threads are easy to manage,
this should not represent any greater effort for feedback.  But then again,
if you are not using flow control, you are also not using any
acknowledgement other than an eventual SCSI status.  If there is no flow
control, this removes resource control and reception acknowledgement.  If
this is not needed, then such flow control should be removed for simplicity
as there then must always be an abundance by design of adequate recourses.
For cases where command sequence is important, CmdSN together with
acknowledgement must remain or only one command at a time can be issued.  I
can envision three modes, a Sequential mode, a Casual mode, and an Exigent
mode where the command is given head of sequence status (with rejection for
feedback simplicity ;).

You seem to be placing many things on the table for reconsideration.  Flow
control, acknowledgement, sequential delivery, and perhaps even a means of
moving a command up the queue feeding SCSI targets.  One command at a time
for sequential assurance is not impossible.  Is flow control needed?  Do you
care if there is any reception acknowledgement if all feedback is in the
SCSI layer?  It would seem in this barebones case, iSCSI adds little to
ensure integrity.  The FC encapsulation places a time stamp within the PDU.
Perhaps a time stamp should be employed as an alternative to sequence
control?  At least after a timeout, you are assured a missing command is
forever gone.  Otherwise, the check was to see that CmdSN is not too small
to ensure it is not a stray.

Doug


> Doug [& All],
>
> Some comments on this thread :
>
> 1) The immediate command feature can be exploited by initiators which
> are not required to provide strict command ordering for their SCSI
> sub-system. (The majority of today's scsi stacks). IOW, all commands can
> be sent with CmdSN=0 to indicate no ordering required.
>
> Any scheme to restrict the number of "immediate commands" that can be
> sent prohibits such a feature and is un-desirable.
>
> 2) The (ExpCmdSN, MaxCmdSN) based acknowledgement and flow control is a
> freebie that targets can use to implement additional flow control.
> There's nothing in the spec that mandates that a target MUST use this
> feature to throttle its window and apply flow control.
>
> Hence, any dependence and assumptions on the existence of CmdSN based
> flow control may be incorrect, since the target may be using QUEUE FULL
> (TASK SET FULL) to apply flow control.
>
> > I also suspect it is a mistake to allow commands
> > to remain trapped within the sequencer during emergency or
> abnormal events
> > signified by the use of these immediate commands.
>
> 3) If the initiator suspects that the CmdSN queue at the target is
> stuck, it can always use a task mgmt command or even an iSCSI login with
> the X (restart) bit to perform a cleanup of stuck commands at the
> target. There's no need to build in implicit assumptions that a CmdSN
> with the immediate flag result in a clean-up of previously pending
> commands at the CmdSN queue of the target.
>
> - Santosh
>
>
> Douglas Otis wrote:
> >
> > David,
> >
> > > > To not support rejects, you are then relying on the target to
> > > > handle these now out of sequence commands.
> > >
> > > Immediate commands are "out of sequence" by definition - immediate
> > > means deliver immediately without regard to CmdSN order.  Only the
> > > target can do this, so I don't see any problem with relying
> on the Target
> > > to do so.
> >
> > The commands that are NOW out of sequence are those commands
> bypassed by the
> > immediate command.  If this immediate command was Rewind, then
> a long list
> > of write commands become invalid as a new tape is now needed for those
> > operations.  You are expecting that the target to handle a long list of
> > invalid commands made possible by the iSCSI sequencer.
> Commands invalidated
> > by immediate commands become out of sequence commands.
> >
> > I'll reserve further comment until Julian has made some progress in
>
> >
> > Forgive my advocacy, but if one does not attempt to support a
> position, then
> > the subject is not explored.
> >
> > Doug