RE: (iSCSI) A question on Zero Copy

[Stephen Byan <Stephen.Byan@quantum.com>]

Randall R. Stewart [mailto:randall@stewart.chicago.il.us] wrote:

> Does the iSCSI layer want:
> 
> A) Plain Zero copy, where the upper layer (iSCSI) asks
>    to read the next available "message" from the wire
>    into a buffer passed to the transport by iSCSI? 
> 
> <OR>
> 
> B) A directed Zero Copy, where the upper layer (iSCSI) asks
>    to read a particular request to a specific buffer?


I think most folks implementing iSCSI want class B zero copy, but it is
restricted to the case of solicited data. Commands and status can be class A
zero copy, or even just copied. 

I don't know what people are thinking about unsolicited data; it seems to me
that it must be buffered anonymously, and thence copied, but the
resource-poor environments with which I am familiar would opt not to support
unsolicited data at all.

It's possible to imagine iSCSI implementations that use another kind of
zero-copy, where the iSCSI application simply lives with a scatter-gather
list of anonymous buffers allocated by the network stack. But I think it's
rather hard to implement iSCSI application code on top of the indirection of
scatter-gather lists. It's much easier to think about your [file system|disk
controller] cache blocks as named, contiguous regions of (possibly virtual)
memory, rather than a random collection of bits of anonymous buffers. I
think the anonymous buffer approach also has a memory utilization penalty,
and so is not too good in memory-constrained environments. So I vote for
class B zero-copy, which lets my application manage memory as named
contiguous buffers.


I haven't the faintest idea how to achieve class B zero copy, without
putting the entire fast-path TCP processing and some of the iSCSI processing
into hardware state-machines running at wire-speed. 

Absent such wire-speed parsing of the headers, I think we're really talking
about a "copy-once" approach on receive, where the packets land in anonymous
buffers (possibly located on the ethernet PCI adapter), and then software
(possibly running on a processor located on the ethernet PCI adapter) parses
the IP, TCP, and iSCSI headers and then sets up a hardware DMA engine to
copy the payload to a buffer in main memory, and simultaneously perform the
checksum checking. Think of an Alteon Tigon ethernet chip on steriods,
running the TCP/IP fast-path code and some iSCSI application-specific code.

I'd appreciate comments, critiques, and info on other approaches to the
problem :-)


Regards,
-Steve

Steve Byan
<stephen.byan@quantum.com>
Design Engineer
MS 1-3/E23
333 South Street
Shrewsbury, MA 01545
(508)770-3414
fax: (508)770-2604