RE: Command Queue Depth (was asymmetric/Symmetric)

[Jim McGrath <Jim.McGrath@quantum.com>]

I agree that BB credit has nothing to do with commands per se, but
illustrates the problem we have had with deciding on policies for the
distributed allocation of device resources over multiple initiators.  In my
postscript I noted that the same problems have arisen on command queue (how
many queue slots do you get), with similarly no satisfactory solution.

On FC-AL, the latency to get an initial credit is typically measured in us
for a couple of reasons.  First, much of that logic has been automated in
the interface hardware (indeed, the major source of delay is typically the
elasticity buffer, which is not store and forward and so has a very low
latency compared to many switches or routers).  Second, the distances are
very small (e.g. hundreds of meters), so both transmission time and the
opportunity for intervening devices to increase latency is lower than in the
general internet world.

So generally 2 credits (of 2Kbyte frames) is enough to cover the latency and
get you into streaming.  If the latency of the system was measured in ms,
then a transport on a Gibt wire would require more like 50 initial credits
(or more) to cover the latency.  Unless we are designing for a low latency
environment for the exchange of credits (like those where FC-AL are used),
then we probably need to allocate so much buffer space that it becomes
difficult to promise initial credits to a lot of potential initiators.

Jim

PS general Fibre Channel (e.g. with switches and the like) is a bit
different.


-----Original Message-----
From: Matt Wakeley [mailto:matt_wakeley@agilent.com]
Sent: Wednesday, September 06, 2000 8:23 PM
To: ips
Subject: Re: Command Queue Depth (was asymmetric/Symmetric)


Jim,

I agree that FC has tried (unsuccessfully) to address this command queue
allocation problem.

However, the "login bb credit" mechanism in FC does not address the command
queue depth issue at all.  BB credit is used to receive commands and/or
data,
and the target has no clue in advance what is coming.  BB credit is just
there
to ensure that there is a lowest layer buffer available to receive the FC
frame
(as opposed to dropping it on the floor if there is no "mac" buffer, like
ethernet does).  It does not mean the command queue has any room for the
frame.

At one time, there was a big push to have "data" credits and "command"
credits
to take care of this problem, but it couldn't be made to work and be
"backwards
compatible".


> The issue of buffer space allocation for multiple initiators has a long
and
> troubled history in SCSI.  We have never been able to come up with a good
> answer.
>
> Fibre Channel tried to fix this with the notion of "login BB credit" -
when
> you login you get a minimum number of credits you are always guaranteed
when
> you start data transfers.  The problem with this is that storage devices
had
> no realistic ability to discriminate between initiators or to change the
> login BB credit.  In addition, the expectation is that all possible
> initiators would get these credits on login.  So storage devices vendors
> have played it safe and kept this number low (at 0 until recently, now
> around 2).  For iSCSI the number of initial credits you need to "prime the
> pump" until normal data flow is established is probably large (given the
> latencies are higher than in Fibre Channel, especially FC-AL), and the

How is the latency low on FC-AL?? given that you need to arbitrate and win
the
loop, then receive bb credit, before you can send anything?

-Matt

>
> number of potential initiators larger than in Fibre Channel, making this a
> whole lot worse for the storage device.
>
> As soon as we allow the devices to start adjusting these credits, then you
> have the protocol problem of making sure people know when their credits
are
> adjusted and the policy problem of how, who, and when to adjust the
credits.
> Changing everyone's credit when you add a new initiator can get into a
> notification nightmare, although it is "fair."  Any policy brings up all
> sorts of nasty issues regarding fairness vs efficient use of the
> transmission media.
>
> Jim
>
> Note: the same problem has plagued other attempts to allocate device
> resources between multiple initiators, like command queue space.  In
general
> policies with respect to multiple initiators are not really standard in
the
> SCSI world.
>
> -----Original Message-----
> From: Matt Wakeley [mailto:matt_wakeley@agilent.com]
> Sent: Wednesday, September 06, 2000 2:56 PM
> To: ips
> Subject: Re: Command Queue Depth (was asymmetric/Symmetric)
>
> Joshua Tseng wrote:
>
> > James,
> >
> > I agree with others that there may be an issue with the
> > command windowing mechanism in the existing iSCSI spec.  It is like
> > "TCP in reverse", in that the target determines the size of the window,
> and
> > not the initiator as in TCP.  Rather, I believe that everything that
this
> > windowing mechanism is attempting to achieve can be more easily obtained
> > by having the target communicate its buffer size to the initiator at
> > iSCSI login.  It should be the role of the initiator to determine how
> > many commands to put in flight simultaneously, given this input on
> available
> > buffer size from the target.
>
> As more initiators connect to a target, it may need to scale back the
amount
> of
> this buffering it has allocated to each previously logged in initiator (to
> prevent rejecting new logins).
>
> >
> >
> > As far as multiple initiators, could this not be resolved by the target
> > refusing additional logins beyond the number of initiators it can safely
> > support?  Not being a storage expert, this is my best guess/suggestion
> > at how to do it.
>
> I believe John already answered this...
>
> -Matt