RE: iSCSI: No Framing

["Mukund, Shridhar" <Shridhar_Mukund@adaptec.com>]

Dear Colleagues,

PART 1 of 3 : 

The pertinent points that "I" see in Jim's email are:

A. Need to hear from iSCSI chip implementers ...
   The issues that you raise for e.g. in #2&#4 are simply
   circumstantial( See PART 2 ). Answers to those questions 
   unnecessarily call for data flows and implementation 
   details that silicon vendors are not allowed to share 
   in public. While I do not know of many silicon vendors 
   in the multi gig space, clearly one of the competitions 
   I respect, namely Trebia, point to FIM as well. Granted, 
   no matter what, we are going to see several 
   trebia-on-the-other-end and adaptec-on-the-other-end 
   type of cost optimizations.

B. The iWARP/TUF/SCTP under-current ...
   The "only" message of significance in Jim's email is #5. 
   It seems like the iWARP/SCTP/TUF enthusiasts somehow feel
   threatened that FIM-iSCSI will dilute the perceived value 
   proposition of iWARP :-) I for one am an enthusiast of 
   iWARP ideals myself, barring the proposed mechanics. I 
   would love to make a buck or two along with you in 
   building iWARP NICs, "at the right time". In the 
   meanwhile, iSCSI is the flag ship effort that has the 
   unique opportunity to make a dent in enabling IP Storage 
   visions. ( See PART 3 )

My assertions are :
    i) TUF/SCTP/iWARP is ruled out in the near term. The 
       mechanics are unclear as hell.
    ii) FIM is the least intrusive, TCP transparent, means for
       enabling low-cost(power,b/w,latency,memory,space,CPU) 
       RDMA transport of bulk data. 
    iii) I do not see significant technical reasons that 
       merit major changes to the iSCSI draft at this late 
       date.
    iii) Making it OPTIONAL to use, and SHOULD only on send 
       provides a graceful and incremental deployment path 
       for "real":-) providers and users to succeed.

I have personally contributed nothing to the iSCSI effort 
and do applaud the pains that several of you are taking to 
pull it all together. In that very same spirit, I respect
David's decision w.r.t. consensus, whatever that turns out 
to be.

Thanks.

-Shridhar Mukund + Todd Sperry, Siva Munnangi, Dev ...

------------------------------------------------------------
PART 2 of 3 : MUST delete, SHOULD read

Dear Jim,

Congratulations Jim! Seems like when you bowl, pins 
roll, unintentional as they may be. You make a "seemingly" 
well balanced set of arguments and "tactfully" tilt the 
balance towards your chosen side. I would love to be on 
your side ... maybe in another effort :-)

I would like to introduce you to my respectable friend, 
who "vehemently" contradicts you on all accounts. One of 
his numerous quotes goes as follows:
http://groups.yahoo.com/group/rdma/message/486
"Much of today's usage of the Internet and IP networks
is for buffer-to-buffer data transfers, often in the 
form of bulk data ... Gigabit and faster network transfers
incur 'heavy' system resource costs, including both CPU
use and system bus bandwidth, particularly on the
receiving side ... The costs incurred on hosts for protocol
processing and management has 'inhibited' the use of IP
in the high speed bulk data domain. ... Bulk data transfer
is dominated by the costs of copying and validating 
incoming data in order to place it at its ultimate 
destination."

The good friend I quote above is none other than Jim 
Wendt himself!!! I REST MY CASE.

Thanks. 

-Shridhar Mukund

----------------------------------------------------------------
PART 3 of 3: MUST delete, OPTIONAL read

On the lighter side ... since the opponents have "no" technical 
arguments whatsoever against FIM and it is all turning out to 
be a pure political gimmick, I will put on my rusting tin 
politician hat :-)

My dear fellow iSCSI country (wo)men: If our goals are anything 
short of enabling the HP's, Cisco's, uSoft's, IBM's and EMC's of 
the world in globalization of storage for McDonald's and Macy's 
from Kabul to Somalia, via iSCSI, we have lost our identity! 
( \insert APPLAUSE for 13 seconds )We are no more protected by 
the vast oceans between us and other Storage efforts. The 
freedom of iSCSI America is threatened from within by elements 
who will not blink twice when it comes to using the world's most 
potent BOO-bombs ... and grinnn while we end up sifting through 
the rubbles for iSCSI, youSCSI and theySCSI. 
( \insert APPLAUSE for 11 seconds ) Beware of the elements that
sleep with "our" very ideals in their privacy(burkha clad 
though) and yet attempt to destabilize us, only to accomplish 
their mutated agenda.( \insert APPLAUSE for 57 seconds )

No offense folks. I respect each and every one of you and 
especially Jim. I think that he was only attempting to 
question, "are we sure ... should we commit ...". 
I disagree with him anyway!

- the running(actually limping) mate :-) :-) :-)

-----Original Message-----
From: WENDT,JIM (HP-Roseville,ex1) [mailto:jim_wendt@hp.com]
Sent: Tuesday, January 29, 2002 10:47 PM
To: ips@ece.cmu.edu
Subject: iSCSI: No Framing


Perhaps we should discuss the possibility of not 
specifying any framing mechanism (FIM or COWS) in the 
first version of iSCSI.

The arguments for not specifying framing include 
(there may be others as well):
1) The brute force approach of putting TCP receiver 
reassembly memory on the iSCSI NIC will work for both 
1Gbps and 10Gbps. Cost is incurred for memory chips, 
ASIC pins, power, and board space. But, it is a 
feasible approach.

2) I/O bus latency (or bandwidth limitations) could 
mandate inbound buffering (as a 'smoothing' buffer) 
from the iSCSI NIC to the host system. If this buffer 
memory is large enough to have to be off-chip, then 
it will require some minimum number of memory chips 
to provide the necessary bandwidth, and the 
memory/pins/power/space costs will be incurred 
anyway.

3) Very large receive buffering on the NIC is only 
required for high-bandwidth links traversing large 
geographic distances (which may not be the 
predominate case). These specialized implementations 
can cost more (e.g. more memory/power/pins/etc) and 
customers will likely be willing to pay accordingly.

4) Many NICs will likely aspire to be combo 
iSCSI+TOE+Ethernet NICs allowing the host system to 
use a single Ethernet port for all of its network 
communications. The TOE function on this combo NIC 
will already require TCP receive buffering and off-
chip buffer memory to support the existing sockets 
interface receive model.  More importantly, to allow 
senders to assume ownership of the buffer upon return 
from a socket send call, the TOE NIC would need to 
copy application send buffers into NIC memory as 
well.

5) The framing/marker mechanism will be an iSCSI-only 
solution. It is quite possible that the problem will 
be solved via a different, and hopefully more 
general, mechanism for other ULPs.

6) Both FIM and COWS are poor solutions for software 
implementation. COWS requires, at a minimum, the 
sender to read every word in the buffer. FIM either 
imposes additional sender gather processing (iovecs) 
or requires a data buffer copy (on systems that don't 
support HW gather on sends).

7) Unless all senders thoroughly test framing/markers 
now (i.e. unless the framing method is a MUST 
implement), there is potential for future 
interoperability problems as framing/marker receivers 
are deployed in the future.

8) Neither FIM nor COWS is an elegant solution. Are 
we comfortable with the legacy we are creating under 
the umbrella of state-of-the-art IP networked 
storage?

9) Is it essential to build in forward compatibility 
now, or will there be a different solution in the 
10Gbps or 40Gbps timeframe - perhaps involving iSCSI-
2?  Will it be reasonable to update or bridge initial 
1Gbps deployments?


So, it would be good to hear from several iSCSI 
NIC/chip implementors who:
- have or plan to implement FIM or COWS (or some 
other framing mechanism) and take advantage of it to 
minimize demands on on-NIC buffer memory 
bandwidth/quantity.
- believe that all-buffers-on-chip solutions are 
feasible and valid (wrt the points above, including 
#2)
- can quantify the memory/pin/power/space cost 
savings for all-buffers-on-chip solutions

Jim