RE: iSCSI: Flow Control

[Michael Krause <krause@cup.hp.com>]

At 01:52 PM 10/16/00 -0600, GUPTA,SOMESH (HP-Cupertino,ex1) wrote:

>--
>Now in cases where the command cannot be posted to the NIC queue,
>it must be left in another queue in the host which is then processed
>when the condition is removed. The condition will be removed
>when a command status is received (also could be RTT but that will
>be useless if the model assumes interrupting the host - you really
>don't want to interrupt the host on RTT) - and the host is interrupted

This does not require an interrupt.  The design can be such that the NIC 
receiving the status simply pulls the next command should it exist.  The 
concept would be no different than ringing a doorbell to show where work is 
available to be processed by a given NIC.


>In a connection-wide model: The interrupt processing routine checks
>the NICs command posting queue (or equivalent status) and if it had
>been full, knows to check the common queue for more commands. If not,
>then it know there is nothing to do for command posting.

Same is true for the host model.

>In a session-wide model: Update the global location of MaxCmdRn
>(take a lock and release lock and thrash cache if multiple CPUs
>active). Then always have to check is there are commands
>waiting to be posted (again by checking variable and locks etc).
>If yes, then post those commands - repeating the algorithm that
>was used when the upper layer posted a command.

This is one implementation choice but there are other techniques that would 
allow this to occur without such a level of host overhead.

One thing to keep in mind is what rate do these exception conditions occur 
and does that rate justify changing the architecture as you 
propose.  Anyone have any workload data to back a particular direction?

>NOTE: If we feel that the SCSI layer will generate commands faster
>than the session-wide credit then the session-wide credit will
>cause extra processing. It is much more straightforward to be
>able to post from the top half, then to have to try to post from
>top-half and then actually post from the bottom. If there is
>significant credit issue, then the outbound command queues will
>  be going through starvation at times.
>
> >
> > Each Target NIC will have a poll of buffers to receive
> > asynchronous (non DATA)
> > iSCSI messages.  As each (small) command message is received,
> > it is placed
> > into one of these buffers, processed by common iSCSI and the
> > CDB is passed to
> > the SCSI layer which stores it into its command buffer. The
> > message buffer is
> > then given back to the NIC for further messages.
>
>The question is how much credit are you going to hand out to the
>remote side. If there are N buffers posted per card and M cards, will
>you make a credit of N available (underutlization) or N * M (which
>assumes that the send will send evenly and is risky if there
>is sudden congestion on one or more connections). Also the
>same discussion of the system cost of a calculating and using a
>centralized value of MaxCmdRn applies if arrays have multiple
>processors.

We should avoid conjecture and show data to justify a particular algorithm 
/ approach in this area.  There are many variants on the above that might 
work well and perhaps the algorithms (as suggested before) need to be able 
to be adjusted (modify the command distribution / arbitration policies) 
based on the throughput objectives and performance analysis across a set of 
paths.

>Look at it as an opportunity to differentiate and streamline
>performance than as a burden. It would definitely be a feature
>for multi-port NICs where all the ports used for a session
>are on the same NIC. Saves host CPU cycles thereby improving
>the attractiveness of the solution :-)

If the session is confined to a single NIC, then I don't see the difference 
between a session versus connection focused implementation - the work is 
accomplished at the session layer within the NIC itself which is how I 
would expect it to be done in the first place for such solutions.  However, 
now a session can also span multiple NICs without changing the overall 
architecture but at perhaps a quantified performance hit - the cost / 
benefit of this performance hit relative to the other advantages of a 
session-based solution needs to be evaluated.

Mike