Re: ISCSI: flow control

[Pierre Labat <pierre_labat@hp.com>]

Robert Snively wrote:

> >  >The question here is whether iSCSI should attempt to solve
> >  the command queue
> >  >overflow problem, or allow T10 to deal with it.  Other
> >  protocols (fibre
> >  >channel) already deal with it in the "ugly" fashion -
> >  perhaps so should
> >  >iSCSI.  What we need consensus on is whether or not iSCSI
> >  is going to deal
> >  >with this.
> >
> >  The "ugly" mechanism is undesirable for a variety of reasons
> >  including fast
> >  convergence / recovery upon data loss.  A credit system is simple to
> >  implement and provides a way to bound this problem and the
> >  implementation
> >  complexity (whether in software or hardware or if operating
> >  across a set of
> >  adapters).   With a single connection implementation, this
> >  is trivial to
> >  accommodate and future multi-port or multi-adapter solutions
> >  can take
> >  advantage of it as you note with minimal effort.  Given this
> >  does not
> >  really impact a single connection solution, what compelling
> >  reason is there
> >  for removing its definition / requirement within the spec at
> >  this point?
>
> Actually, the ugly problem is not made much less ugly by having
> a credit system.  The problem is that the command receipt resources
> of a "target" are oversubscribed during periods of high activity on
> a properly configured system simply because of the statistical
> characteristics of storage access.  The command resources are also
> shared among multiple sessions.  The only way to guarantee credit
> is to share the credit out among the multiple sessions.  This
> necessarily leaves a lot of unused resources lying around for
> sessions that are not very busy at a given instant and unnecessarily
> limits the capabilities of those sessions that are busy and would
> otherwise be able to make constructive use of those resources.
> The result is higher latencies and lower throughputs than you would
> hope.  What you really need is dynamic credit allocation, which will
> always leave the possibility that a particular session may find
> itself oversubscribed and forcing busy indications.  Which is effectively
> the mechanism SCSI uses today.

Robert,

As you say, the target can inform the initiator whenever it wants, based on
its
own policy of resources management, that it can't handle no more
command. It has the advantage to be dynamic and to take into account
all the initiators.
However, this mechanism has the following weaknesses:
1) to inform the initiator to stop sending command, the target
     drops commands (and data). It implies retransmission
    It's a waste of bandwidth.

2) the initiator after detecting [several] TASK SET FULL/BUSY status,
must handle blindly the return to the full depth of the command window.
It does that in a slow way. At that time you under use the capabilities
of the LU.
I explain: in a typical SCSI implementation, when a TASK  SET FULL
set occurs, the initiator decrements  the max queue depth
(command on flight for the LU) by one. Then after a number
of commands completed (say N=32) the initiator increments the max
queue depth to return to the initial value.
The initiator has to comeback to the initial queue depth slowly and
cautiously because never the target informs it, the queue depth available
for it.

This behavior looks like a target congestion control mechanism
1) = you drops

2) = it's a kind of slow start

Why stick with this mechanism that on one side waste bandwidth
and on the other side "under use" the capabilities of
a LU because the slow start?

It would be better to add a comand flow control mechanism
that acts before we hit  this target congestion control mechanism.

T10 could add an advertising of a command credit. For example
when the status is returned the target gives a command credit.
Doing that way the target is able to slow down the command
flow without dropping, and then drive the initiator in order
it catches up quickly with the recovered capability/availibility
of the LU for it.

Regards,

Pierre

>
>
> Bob Snively
> Brocade Communications           Phone  408 487 8135
> 1745 Technology Drive
> San Jose, CA 95110               Email   rsnively@brocade.com