Re: New List: rdma@cisco.com: to discuss RDMA

[Michael Krause <krause@cup.hp.com>]

At 04:22 PM 9/28/00 -0500, Stephen Bailey wrote:
> > However, I would support structuring the spec so that an RDMA
> > transport mechanism could be used underneath (I guess that's
> > motherhood).
>
>Not necessarily.
>
>You have to ask the implementors (particularly the hardware
>implementors), what sort of optional RDMA proposal they'd be happy
>with.  My answer is none.  It's mandatory or not at all.

I disagree.  We had a similar debate within InfiniBand on just this subject 
and it really came down to implementation specific requirements.  The IB 
host channel adapters were required to support all of the RDMA semantics in 
order to insure the base functionality is ubiquitous but the target channel 
adapters (think of this as a NIC) were allowed to support all, some, or 
none of the RDMA operations.  For example a storage adapter might accept 
SEND operations which contain the SCSI command. It could then issue RDMA 
READ operations to obtain the data from the source.  The same would apply 
to issuing RDMA WRITE operations in response to a data read 
operation.  This can be implemented in a fairly thin resource adapter with 
minimal complexity and one might say is quite similar to any number of 
existing SCSI or FC implementations.  The benefit of RDMA will vary 
depending upon how much functionality and resources are available on each 
side of the communication and whether the usage is uni- or bi-directional.

It should be noted that RDMA and many of the storage implementations still 
"lock" up resources for potentially long periods of time which can lead to 
application-level resource contention and throughput degradation.  This is 
a problem that no one has really addressed - they've mitigated the impact 
but as bandwidths increase the impacts of such resources caches and 
application cache contention will worsen.  I don't view RDMA or any of 
these implementations as panacea solutions for this overall resource usage 
problem.

>The reason for using RDMA is to make the implementation of iSCSI
>easier in hardware.  If there are implementations which do not support
>the RDMA protocol, then your hardware implementation will have to
>support both the `easy path' (using RDMA) and the `hard path' (no
>RDMA).  If you have to implement the hard path anyway, there's no
>point in implementing the easy path.

Debatable.  Obviously, iSCSI does require some form of option negotiation 
at session / connection establishment and RDMA is one of the options that 
should be included in this negotiation.  The benefits of RDMA are 
associated with the ability to reduce the amount of buffering required 
within a solution at the various points along the way and deliver 
zero-processor copy placement and control.  These can be implemented in 
other ways as others have pointed out and thus the cost / benefit needs 
will vary.

It should be noted that other protocols (e.g. SVP) and perhaps the ability 
to implement higher-level functionality may be more easily implemented in 
the future if a general-purpose RDMA solution is defined.  This is why 
creating such a solution and enabling iSCSI to take advantage of such 
implementations is critical at this point in time.

>The argument that you could make the hard path infrequent and
>implement it in software doesn't wash in this case.  It just takes one
>implementation that doesn't do RDMA to slow your system to a crawl,
>and the competitor who only implemented the non-RDMA path makes you
>look like a fool.

Let the market decide whether that one implementation should survive or 
not.  Most likely it will die as others will have been smarter in their 
designs and functional selection.

>Fundamentally, RDMA has to be either adopted or punted.  Of course,
>I'm happy to have somebody prove this statement wrong.

The issue is one of the level of gray not black and white.

Mike