RE: iSCSI: No Framing

["Sriram Rupanagunta" <sriramr@aarohicommunications.com>]

> 
> Perhaps we should discuss the possibility of not 
> specifying any framing mechanism (FIM or COWS) in the 
> first version of iSCSI.
> 

I agree with all your comments, and this seems to be the
reasonable approach to take. Brute force method, as you 
pointed out correctly, may be acceptable for NICs/Off-load engines
working over long-fat pipes.

-Sriram
Aarohi Communications, Inc. 


> The arguments for not specifying framing include 
> (there may be others as well):
> 1) The brute force approach of putting TCP receiver 
> reassembly memory on the iSCSI NIC will work for both 
> 1Gbps and 10Gbps. Cost is incurred for memory chips, 
> ASIC pins, power, and board space. But, it is a 
> feasible approach.
> 
> 2) I/O bus latency (or bandwidth limitations) could 
> mandate inbound buffering (as a 'smoothing' buffer) 
> from the iSCSI NIC to the host system. If this buffer 
> memory is large enough to have to be off-chip, then 
> it will require some minimum number of memory chips 
> to provide the necessary bandwidth, and the 
> memory/pins/power/space costs will be incurred 
> anyway.
> 
> 3) Very large receive buffering on the NIC is only 
> required for high-bandwidth links traversing large 
> geographic distances (which may not be the 
> predominate case). These specialized implementations 
> can cost more (e.g. more memory/power/pins/etc) and 
> customers will likely be willing to pay accordingly.
> 
> 4) Many NICs will likely aspire to be combo 
> iSCSI+TOE+Ethernet NICs allowing the host system to 
> use a single Ethernet port for all of its network 
> communications. The TOE function on this combo NIC 
> will already require TCP receive buffering and off-
> chip buffer memory to support the existing sockets 
> interface receive model.  More importantly, to allow 
> senders to assume ownership of the buffer upon return 
> from a socket send call, the TOE NIC would need to 
> copy application send buffers into NIC memory as 
> well.
> 
> 5) The framing/marker mechanism will be an iSCSI-only 
> solution. It is quite possible that the problem will 
> be solved via a different, and hopefully more 
> general, mechanism for other ULPs.
> 
> 6) Both FIM and COWS are poor solutions for software 
> implementation. COWS requires, at a minimum, the 
> sender to read every word in the buffer. FIM either 
> imposes additional sender gather processing (iovecs) 
> or requires a data buffer copy (on systems that don't 
> support HW gather on sends).
> 
> 7) Unless all senders thoroughly test framing/markers 
> now (i.e. unless the framing method is a MUST 
> implement), there is potential for future 
> interoperability problems as framing/marker receivers 
> are deployed in the future.
> 
> 8) Neither FIM nor COWS is an elegant solution. Are 
> we comfortable with the legacy we are creating under 
> the umbrella of state-of-the-art IP networked 
> storage?
> 
> 9) Is it essential to build in forward compatibility 
> now, or will there be a different solution in the 
> 10Gbps or 40Gbps timeframe - perhaps involving iSCSI-
> 2?  Will it be reasonable to update or bridge initial 
> 1Gbps deployments?
> 
> 
> So, it would be good to hear from several iSCSI 
> NIC/chip implementors who:
> - have or plan to implement FIM or COWS (or some 
> other framing mechanism) and take advantage of it to 
> minimize demands on on-NIC buffer memory 
> bandwidth/quantity.
> - believe that all-buffers-on-chip solutions are 
> feasible and valid (wrt the points above, including 
> #2)
> - can quantify the memory/pin/power/space cost 
> savings for all-buffers-on-chip solutions
> 
> Jim
>