Re: TCP Framing (considered helpful?)

[Stephen Bailey <steph@cs.uchicago.edu>]

John,

> I think we must depend on Markers to insure that everything can operate at
> top speed, and at the lowest cost.

A key question is whether markers actually ensure that everything
operates at `top speed, and at the lowest cost'.

Matt thinks so.  I (and, presumably those who wrote the framing
document) think not.

My issue is not even with `lowest cost'.  I don't believe markers will
allow you to run at top speed.  Specifically:
  1) I doubt the feasibility of implementing the control required for
     an eddy buffer (where you store data you can't place) at 10G.
     Admittedly, the validity of this claim can't really be assessed
     without actually working the implementation, so for 99% of the
     list participants (myself included) this is a `yes it is, no it
     isn't' point.
  2) an eddy buffer solution requires some substantial speed-up in
     both the NIC data path, and MOST IMPORTANTLY: the host bus.  In
     order to unload the eddy buffer while still handling incoming
     traffic at line rate, clearly the host bus bandwidth must be >
     line rate.  

I know of at least one general purpose framed solution operating at
10G which has been available for >3 years (SGI's GSN/ST/XIO NIC).  I'm
sure there are others.

I can't imagine there's any argument that a framed solution would be
voted `most likely to run fast and be cheap'.  Every storage network
and cluster interconnect has been designed that way since antiquity.

The key tradeoff involves the OS vendors, and I'm wondering why we're
speaking for them.  The question IS, how much more work is it to
introduce TCP framing over and above what is required to insert iSCSI
into their network framework.  My experience from writing NIC and
storage drivers for many commercial UNIX-family OSes is:
  1) it's an easy and well defined process to insert a new SCSI
     transport driver into the SCSI stack.
  2) it's hard and poorly defined process to insert ANYTHING into the
     network stack.

Networking has historically been a user-mode activity.  Architected
services are only provided to user mode programs.  Kernel clients have
been few and far between and so are handled on a case-by-case basis.
For example NFS.  Every OS has hacks to make NFS run fast, but they
are not stable interfaces for general purpose use.

Even Solaris' SysV-derived STREAMS stack, which is intended precisely
to provide flexible, crisp interfaces for kernel network clients, does
not document the relevant (IP stack) intermodule interfaces.

I know that there are more and more kernel network clients, but they
are coming either on fluid platforms (e.g. linux), in which case the
argument of `it'll take too long to get OS support' doesn't apply, or
they are vendor-supplied, in which case a performance iSCSI solution
in ANY form may take a while, and the choice of framing or markers
isn't going to make a difference.

I can't say squat about the architecture of Winsock, but the fact that
there is a Microsoft author of the framing proposal who seems very
serious about supporting framing and RDMA as quickly as possible
suggests that framing support should be available on Windows very
soon.

Steph