RE: ISCSI: flow control

[Robert Snively <rsnively@Brocade.COM>]

>  > Actually, the ugly problem is not made much less ugly by having
>  > a credit system.  The problem is that the command receipt resources
>  > of a "target" are oversubscribed during periods of high activity on
>  > a properly configured system simply because of the statistical
>  > characteristics of storage access.  The command resources are also
>  > shared among multiple sessions.  The only way to guarantee credit
>  > is to share the credit out among the multiple sessions.  This
>  > necessarily leaves a lot of unused resources lying around for
>  > sessions that are not very busy at a given instant and 
>  unnecessarily
>  > limits the capabilities of those sessions that are busy and would
>  > otherwise be able to make constructive use of those resources.
>  > The result is higher latencies and lower throughputs than you would
>  > hope.  What you really need is dynamic credit allocation, 
>  which will
>  > always leave the possibility that a particular session may find
>  > itself oversubscribed and forcing busy indications.  Which 
>  is effectively
>  > the mechanism SCSI uses today.
>  
>  Robert,
>  
>  As you say, the target can inform the initiator whenever it 
>  wants, based on
>  its
>  own policy of resources management, that it can't handle no more
>  command. It has the advantage to be dynamic and to take into account
>  all the initiators.
>  However, this mechanism has the following weaknesses:
>  1) to inform the initiator to stop sending command, the target
>       drops commands (and data). It implies retransmission
>      It's a waste of bandwidth.

This is one of many reasons I do not believe that sending immediate
data is a constructive exercise.  Having to resend a few bytes of
command information on those rare occasions that your system
is so underconfigured that you run out of command queuing resources
doesn't seem to me to be a major problem.  However, running out of
data buffering (as opposed to command queuing) resources could be a 
frequent occurrence and should be avoided by using the appropriate
Ready To Receive indication from the device.

>  
>  2) the initiator after detecting [several] TASK SET FULL/BUSY status,
>  must handle blindly the return to the full depth of the 
>  command window.
>  It does that in a slow way. At that time you under use the 
>  capabilities
>  of the LU.

My point is that a properly implemented system does not do this.  Instead,
it uses an optimistic start up algorithm.  By reserving
one slot for each likely initiator, the SCSI device is always able
to accept at least one command from that initiator.  Dynamically
allocating the remainder of the command queuing resources allows any
combination of additional commands from any initiator to be queued
up until the queue is full (except for the reserved slots for initiators
with no activity).  

If one of those initiators then starts up, it starts by optimistically
sending all its commands.  Most of the time, there will be plenty of
space and the target will cheerfully absorb the commands.  Sometimes, the
queue will be all the way full and only one command will be absorbed.
If that is the case, the remaining commands are returned with a queue
full indication.  Eventually, the one queue command from that initiator
will be completed and sent back to the initiator.  At that time, it
makes the assumption that lots of other commands have also been cleared
out by other initiators, and sends to the target all the commands it
wants to send.  By this time, there will again usually be plenty of
space.  Some implementations may have a sense of how much queue space
is typically available and will limit their first optimistic delivery
of requests to some number, but if demand is high for a particular
initiator, that initiator will push the numbers beyond that limit
very quickly.

If you have a situation where an initiator is frequently encountering
no command queuing resources at a target, you have an improperly 
configured system and must either:

	a)  sacrifice performance (not total throughput, but individual
		observed latency);

	b)  reduce the number of initiators and/or the load provided by
		each; or

	c)  increase the capability of the target.

Note that this has nothing to do with congestion management anywhere, 
since all these packets are very small.