RE: iSCSI: Flow Control

["Y P Cheng" <ycheng@advansys.com>]

> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> David Robinson
> For any high speed NIC, it must be able to move data
> into system memory at wire speeds, either directly or through
> some speed matching buffers. If there is any processing that
> cannot be done between packets, the data must get moved to
> a large pool.
>
> This is a general how to design a high performance adapter, but we
> should not add features to compensate for bad designs.

Dave,
You have put it very well on the essence of designing a high performance
NIC.  In fact, all the flow control issue and head-of-queue blocking
problems are non-issues for a properly designed iSCSI adapter.  I described
the NIC design in several postings earlier.  The messages seem have lost.
Let me summarize the important points herein again:

1) An iSCSI adapter driver receives requests from a SCSI-class driver.  The
requests are disk reads and writes, Inquiry, mode select, or task management
commands, etc.
2) The iSCSI driver for an initiator will use TCP to make a connection to a
well-known port to get the endpoint address of a target node.  There are
multiple connections to multiple target devices.  Multiple connections to
one device is possible.
3) The iSCSI target driver listens to a well-known TCP/iSCSI port for
incoming connection requests.  There is one dedicated connection to each
initiator.  Multiple connections to one initiator is possible.  In such
case, command ordering needs be specified.
4) With its DMA function, an iSCSI NIC always runs faster than media wire
speed.  It uses speed matching FIFOs to accommodate speed variations of the
host system bus.  If a NIC runs slower than the wire speed, then,
statically, packet dropping is guaranteed, because there is no control of
how many incoming TCP segments at any instant.
5) An iSCSI adapter implements all three IP/TCP/iSCSI layers by creating the
proper headers for the outgoing segments and parsing the headers of the
incoming segments.
6) A SCSI request to the iSCSI adapter contains the SCSI CDB and data buffer
descriptor provided by application software plus other needed parameters.  A
iSCSI NIC adapter operates directly from the buffers of the application
software.
7) The application on the initiator locks down the need buffers before
making a request to an iSCSI NIC.
8) For each connection, the application software on a SCSI target provides
required buffers for incoming commands and immediate data.  The amount of
memory limits the number of commands and amount of immediate data.
9) Similar to an initiator, the target locks down the buffers as well.  If
an incoming TCP segment has no waiting buffer, the segment is simply thrown
away.  A unsolicitated command or data PDU always has a waiting buffer.
10) Like an initiator, the target NIC transfers directly to/from the
application buffer from/to the media wire through the speed match FIFO.
11) An initiator will not send more commands allowed by the target to avoid
overflow that causes out-of-order execution.
12) Both initiator and target use a large exchange table to track requests
on all connections for different targets and initiators.  An exchange table
entry is created to start a new SCSI request and destroyed after completing
the request.
13) A fully qualified exchange ID is the endpoint address plus the CmdRN.
14) While the command and status PDUs are process in order from queues, the
data PDUs are processed on demand without the deadlock problem.  A large
data transfer will be broken into sequences to avoid blocking of small
command and status PDUs.
15) An iSCSI NIC must handle missing and duplicated TCP segments.  It also
implements congestion avoidance.  Multiple PDUs can be in one TCP segment.
16) An iSCSI NIC supports both IP and iSCSI protocols.  (Other protocols
like VI can be added with more microcode.)
17) An iSCSI NIC adapter is designed to handle 50,000 IOs per second.  When
the roundtrip time of a single IO is 100 msec, the adapter either keeps more
commands in the pipe or send more commands to other iSCSI devices.
Therefore, it tries to process on IO every 20 microseconds.  In the near
future, it is down to 10 microseconds per IO.
18) An iSCSI NIC breaks up large data transfers into segments and tries to
fill the pipe on a network with long latency with large number of segments.
It manages the ACK's to detect lost and duplicated segments.

As everyone can see, for those who design a competitive iSCSI NIC adapter, a
set of rules different from those discussed by this WG is followed.