RE: Command Queue Depth (was asymmetric/Symmetric)

[julian_satran@il.ibm.com]

Charles,

Command ordering was very much on our minds. With the symmetric model -
this is why we have the command numbering scheme and the "ordering window".
With the symmetric model - as commands travel on one TCP connection only
TCP
will guarantee that they arrive in order (TCP is a streaming protocol and
guarantees ordering).

Julo

Charles Monia <cmonia@NishanSystems.com> on 07/09/2000 23:52:22

Please respond to Charles Monia <cmonia@NishanSystems.com>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   "Ips (E-mail)" <ips@ece.cmu.edu>
Subject:  RE: Command Queue Depth (was asymmetric/Symmetric)




Hi Julo:

> -----Original Message-----
> From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com]
> Sent: Thursday, September 07, 2000 6:11 AM
> To: ips@ece.cmu.edu
> Subject: RE: Command Queue Depth (was asymmetric/Symmetric)
>
>
>
> Dear colleagues,
>
> Although the windowing mechanism in iSCSI-01 may seem to be
> there to solve
> a queueing issue
> it is mainly meant to limit the buffering space for commands
> that await
> "de-skewing".
> We assume that execution queue-lengths, policy etc. are
> beyond the scope of
> transport.
>
> As for SCSI queue length I assumed that the busy or queue full status
> followed by an Asynch Event
> message indicating readiness is the mechanism provided by
> SCSI to regulate
> the command flow.
>
> It is hard to imagine that give the variable life-time of
> SCSI commands and
> the
> opaque nature of the resources required to execute them  that
> the transport
> has
> to help in this area.
>

While this issue has been discussed at some length in the past, as Jim
McGrath stated, I believe the debate ought to be reopened (although we may
end up reaching the same conclusion as before).

As Ralph Weber pointed out, the SCSI model is to discard the command,
return
status, retrieve the next command in the transport pipeline and continue
processing.  The initiator is not notified when processing resumes. If
there
are many commands in flight, as there could be in an IP environment, and
target resources free up in the meantime, the result is commands processed
out of order.

Historically, such a lapse in command ordering was not seen as an issue for
the following reasons:

a) Strict ordering was not required by the most commonly deployed device
types (disks and tapes). Due to the nature of disk traffic, a simple retry
mechanism was deemed sufficient to recover from these errors. Since legacy
streaming devices, such as tapes, did not support command queuing, command
ordering considerations were not a factor there either.

b) Transport delays over storage interconnects were small, so not many
commands were apt to be in flight. i.e.. The window for such errors was
very
small.

c)  The resource guarantees needed for a loss-avoidance mechanism in the
target adversely effected device cost, especially at the high-volume,
low-end of the market.

Given the above considerations, there was little support within the storage
community for measures addressing this issue.

If we now believe that the iSCSI environment changes the rules, I believe
the interconnect protocol can provide useful assists, such as:

a)  On a command overflow condition, have the iSCSI target flush the
command
pipeline by returning status and discarding all subsequently received
commands until a host acknowledgement is received.

b)  Implement some sort of credit-based mechanism for overflow-avoidance.


Comments?


<Stuff deleted>

> Jim McGrath <Jim.McGrath@quantum.com> on 07/09/2000 06:06:40
>
> Please respond to Jim McGrath <Jim.McGrath@quantum.com>
>
> To:   "'Matt Wakeley'" <matt_wakeley@agilent.com>, ips
> <ips@ece.cmu.edu>
> cc:    (bcc: Julian Satran/Haifa/IBM)
> Subject:  RE: Command Queue Depth   (was asymmetric/Symmetric)
>
>
>
>
>
> The issue of buffer space allocation for multiple initiators
> has a long and
> troubled history in SCSI.  We have never been able to come up
> with a good
> answer.
>
> Fibre Channel tried to fix this with the notion of "login BB
> credit" - when
> you login you get a minimum number of credits you are always
> guaranteed
> when
> you start data transfers.  The problem with this is that
> storage devices
> had
> no realistic ability to discriminate between initiators or to
> change the
> login BB credit.  In addition, the expectation is that all possible
> initiators would get these credits on login.  So storage
> devices vendors
> have played it safe and kept this number low (at 0 until recently, now
> around 2).  For iSCSI the number of initial credits you need
> to "prime the
> pump" until normal data flow is established is probably large
> (given the
> latencies are higher than in Fibre Channel, especially FC-AL), and the
> number of potential initiators larger than in Fibre Channel,
> making this a
> whole lot worse for the storage device.
>
> As soon as we allow the devices to start adjusting these
> credits, then you
> have the protocol problem of making sure people know when
> their credits are
> adjusted and the policy problem of how, who, and when to adjust the
> credits.
> Changing everyone's credit when you add a new initiator can get into a
> notification nightmare, although it is "fair."  Any policy
> brings up all
> sorts of nasty issues regarding fairness vs efficient use of the
> transmission media.
>
> Jim
>
> Note: the same problem has plagued other attempts to allocate device
> resources between multiple initiators, like command queue space.  In
> general
> policies with respect to multiple initiators are not really
> standard in the
> SCSI world.
>
>
> -----Original Message-----
> From: Matt Wakeley [mailto:matt_wakeley@agilent.com]
> Sent: Wednesday, September 06, 2000 2:56 PM
> To: ips
> Subject: Re: Command Queue Depth (was asymmetric/Symmetric)
>
>
> Joshua Tseng wrote:
>
> > James,
> >
> > I agree with others that there may be an issue with the
> > command windowing mechanism in the existing iSCSI spec.  It is like
> > "TCP in reverse", in that the target determines the size of
> the window,
> and
> > not the initiator as in TCP.  Rather, I believe that
> everything that this
> > windowing mechanism is attempting to achieve can be more
> easily obtained
> > by having the target communicate its buffer size to the initiator at
> > iSCSI login.  It should be the role of the initiator to
> determine how
> > many commands to put in flight simultaneously, given this input on
> available
> > buffer size from the target.
>
> As more initiators connect to a target, it may need to scale back the
> amount
> of
> this buffering it has allocated to each previously logged in
> initiator (to
> prevent rejecting new logins).
>
> >
> >
> > As far as multiple initiators, could this not be resolved
> by the target
> > refusing additional logins beyond the number of initiators
> it can safely
> > support?  Not being a storage expert, this is my best
> guess/suggestion
> > at how to do it.
>
> I believe John already answered this...
>
> -Matt
>
>
>

Charles Monia
Senior Technology Consultant
Nishan Systems Corporation
email: cmonia@nishansystems.com
voice: (408) 519-3986
fax:   (408) 435-8385