iSCSI/VI/TCP

[Costa Sapuntzakis <csapuntz@cisco.com>]

I have sent out a proposal (draft-csapuntz-iscsivi-00.txt) that argues
for running iSCSI on top of the VI/TCP protocol. I argue that there
are a couple principal reasons for doing this.

    1) Generic RDMA infrastructure. 

    A single NIC that support VI/TCP RDMA acceleration could
    accelerate SCSI, DAFS (http://www.dafscollaborative.org/), Oracle,
    and other message+RDMA protocols w/o the need for specific protocol
    processing in the NIC.

    2)  Reduce the need for TCP reassembly memory 

    Why? For performance, TCP maintains a window at least as large as
    the application buffers awaiting data.  The window buffer must be
    separate memory than the application buffers if the receiver cannot
    figure out which application buffers TCP payloads belongs to. 
    Thus, buffering requirements are potentially up to 2x that of the base
    application.

    - VI/TCP could align its headers at the start of each TCP segment payload
        
    [D] This is not currently well discussed in the spec. However, there
    are a couple things techniques that could be used to indicate 
    alignment. For one, a TCP option could be used to indicate that
    the sender will send aligned packets.

    As added check, each VI frame could end in a CRC. Failure of the CRC
    indicates loss of synchronization. Also, there are some fields in
    the VI/TCP header that should make sense (like the Segment Length <
    MTU). 

    - The VI/TCP header, which appears in each segment, could direct
    the payload  of the segment to the correct application-layer buffer
    (for both messages and data)

    3) More exact flow control

    VI/TCP provides flow control on messages separate from data buffers.
    

It is worthwhile to note that iSCSI could be modified to incorporate 
similar features to VI/TCP and do #2 and #3. 

Others have stated some valid concerns with iSCSI/VI:

>- It adds another protocol layer (VI/TCP)

>- it adds another dependency (VI/TCP is not a standard yet)
>   also heard, don't base new technology on new technology

>- all SCSI class drivers and file systems that I'm aware of operate in kernel
>space.  So what advantage does VI bring when the app must enter kernel space
>to perform the I/O anyway?

RDMA can improve kernel I/O performance too.

>- it adds more complexity to developing the iSCSI accelerator to add yet
>another protocol layer.

To this, I would answer that it allows the construction of a
potentially simpler accelerator that handles more protocols.

> There already is a SCSI/VI.

Our group is looking for the best way to do SCSI over IP. For a long
time, members of this group have though that some kind of RDMA layer
would be valuable for building simpler NICs.  VI/TCP provides just
that sort of RDMA layer.

SCSI/VI was born out of a different set of requirements than iSCSI.
However, it may turn out that analysis of the problem brings us
to a very similar solution.

At the end of the day, this argument comes down to whether folks think
that an RDMA mechanism is the right way to go and what RDMA scheme
over IP is best. A suitably generic RDMA scheme allows innovation in
protocol design, without worrying that the new protocol will be
hopelessly slow in legacy hardware.  It decreases the pressure to load
features into old, hardware-supported protocols instead of introducing
new ones. It destroy an artificial barrier (host overhead) that
prevents IP from scaling to fit all our needs.

-Costa