RE: VI (Was: Avoiding deadlock in iSCSI)

[Michael Krause <krause@cup.hp.com>]

At 06:22 AM 9/13/00 -0700, Stephen Byan wrote:
>Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote:
>
> > I don't think VI/TCP helps at all.  The VI is implemented on
> > top of a TCP
> > stream.  If the TCP stream delivers the iSCSI command to VI,
> > and VI has no
> > place to put it, what is VI going to do?  It has to block the
> > TCP stream, and
> > that in turn will block and "RDMA" from occurring.
>
>If I understand correctly, VI/TCP has a credit-based flow control mechanism
>on its message queues (which would be used to implement the iSCSI command
>reception queue), and so the initiator would never send an iSCSI command
>without a target buffer in which to store it. So RDMA cannot block on
>commands in the TCP stream.

InfiniBand has this as well.  One exchanges credits for the number of 
outstanding receive buffers that can be posted.  If one exceeds the credit, 
the responder returns a RNR-NAK (receiver not ready) with a time to wait 
before trying again.  This is completely independent of RDMA.

RDMA is implemented either as a set of READ (fetch data from address X and 
deposit in my local  memory - very nice for thin devices) or WRITE (send 
this data to address Y from my local memory).  The memory must be 
registered, i.e. pinned / mapped, to avoid having to perform the V-P 
translations and to insure it is present for DMA operations.  The advantage 
to using RDMA is that it pays the price for buffer management during the 
heavy weight registration process up front and the subsequent operations 
are therefore much lighter weight.  It also does not have a flow control 
issue for RDMA WRITEs since if a command is allowed to target a given 
memory area, it can always target that memory.  For RDMA READs things are a 
bit more complex w.r.t. ordering rules, resource management (need the 
ability to control how many are outstanding since the responder must be 
able to store the read command and thus requires some form of a command 
queue).  In general, there are many benefits to RDMA support and I would 
like to see TCP have a RDMA capability but they are not required to make 
iSCSI operate (might improve some operations though).

Mike