Re: iSCSI: Flow Control

[Matt Wakeley <matt_wakeley@agilent.com>]

Somesh,

I still don't understand what you are trying to solve.

With the iSCSI session wide command credit method, there is a portion of the
iSCSI layer that sits right below the SCSI layer.  It receives the commands
from the SCSI layer and passes the results of each I/O from each NIC back to
the SCSI layer. The MaxCmdRn indicates how many commands the target (as a
whole) can "buffer". The iSCSI layer will "scatter" the commands to the NICs
until it has used up the MaxCmdRn buffers. Each NIC, once iSCSI has posted a
command to it, will attempt to send the command as long as the TCP window is
open. Practically every message sent from the target to the initiator contains
the new MaxCmdRn.  Each in initiator NIC that receives a message passes this
(new) value to the common iSCSI.  This value does NOT have to be sent to every
other NIC, since once a command is posted to a NIC, it is committed to send
it.

Each Target NIC will have a poll of buffers to receive asynchronous (non DATA)
iSCSI messages.  As each (small) command message is received, it is placed
into one of these buffers, processed by common iSCSI and the CDB is passed to
the SCSI layer which stores it into its command buffer. The message buffer is
then given back to the NIC for further messages.

"GUPTA,SOMESH (HP-Cupertino,ex1)" wrote:

> Yes I am trying to describe the synchronization pts and software
> intervention caused by a session wide flow control model

But I still don't understand the "problem" that the credit per connection
solves over the credit per session model.

In your description, the initiator still "scatters" the commands to the NICs,
then the NICs have the burden of trying to figure out if they can send the
command or not.  Furthermore, if some NICs have open TCP windows, but don't
have command credit, the command can't be sent.

In the iSCSI session wide credit model, the initiator will not post commands
to any NIC if it doesn't have credit.  Any commands posted to a NIC will be
sent as long as it's TCP window is open.

> 1. Post a large enough number at each NIC. OK. The window open up
> (indicated through a new MaxCmdRn received on one connection). This
> value now must be communicated to the other connections, so that
> they can not be flow controlled also. Or the new value must be
> received on each connection.

As I indicated above, the goal is to not overflow the SCSI command buffer, so
the command is not discarded causing a lot of error recovery.  A command CDB
is only 16 bytes.  It does not make sense to allocate 16 byte buffers to NICs
for command reception. As I indicated above, the NIC receives the message, the
iSCSI layer strips out the CDB and hands it to SCSI, then reposts the message
buffer to the NIC.

> Also since you have posted a large enough number at each NIC,
> you are really not having any benefit at all from the session-wide
> value - what is the advantage?

Having a session wide MaxCmdRn allows the initiator to stop sending SCSI
commands, while still enabling non command messages to be sent.  They are
received by each NIC and passed to iSCSI for processing, but since they are
not
passed up to SCSI, nothing is overflowed.


> 2. Have the NICs grab them from a pool through an atomic bus
> transaction. That has got to be tougher to implement than it
> looks, and the bus performance issues due to the need to maintain
> ordering etc?

As indicated above, each NIC passes the iSCSI messages to a central iSCSI
message processor that sends the appropriate SCSI messages to SCSI.

-Matt