RE: TCP speed

["Y P Cheng" <ycheng@advansys.com>]

Julo wrote:
>Our experience is the same. TCP is FAST.
>The only remaining trouble is memory copy from TCP buffers
>to application buffers. Unless
>handled properly this may slow you down considerably.

The issue is not the TCP memory to memory copy speed, it is the latency time
of receiving TCP acknowledges.  Between two endpoints of New York and Los
Angeles, latency is in milliseconds if not in seconds.  On a one-gigabit
network, for each millisecond there are 100K of data, or 66 1.5K datagrams
being transferred.  In fibre channel, there is this EE-credit, End-to-End.
If the sending party has 10 EE credits, it can't send more than 10
datagrams.  EE-credit manages the TCP sliding window currently discussed in
iSCSI.  After sending 10 datagrams, one must wait for acknowledges that may
take several hundred milliseconds to come.

I do believe TCP is a wrong protocol for iSCSI.  A SCSI request from an
initiator is inherently acknowledged by its response from a target.
Therefore, UDP for iSCSI is a better choice.  NFS is implemented on UDP.

The real question is how many datagrams can we pipe to a receiving node
without causing overrun.  This issue is being addressed by the number of
receiving buffers inside the NIC adapter that moves data directly to the
buffers of an application (known as remote DMA in VI for InfiniBand, or
physical writes in 1394, or exchange handling in fibre channel.)  It is a
challenge for people who design the NIC adapter hardware which must keep up
with the speed of connecting media at two gigabits per second today and 10
gigabits next year.

The TCP memory to memory copy speed is irrelevant if we have a NIC adapter
that can transfer data directly to buffers of application software.  If we
don't have such an adapter, it is an impossible task in trying to define a
protocol regulating hundreds of SCSI target devices returning data to a
single SCSI initiator at the same time.  If we slow it down by allowing only
one SCSI target to return data at a time, then, with the long latency time
between two nodes, iSCSI does not stand a chance in the world of OC-192,
gigabit Ethernet, Fibre Channel, InfiniBand, and even 1394B.