RE: iSCSI Urgent Pointer

["Douglas Otis" <dotis@sanlight.net>]

John,

> I have been watching this discussion for some time now.  The following is
> what I gleam out of the discussions.
>
> First, we must talk about many sessions that come to the same
> target.  Many
> of the messages, on the reflector,  take a SINGLE Session view, and make
> arguments that say things like  "it will slow down anyway during a
> recovery, so why do it the Urgent Ptr.  Instead, you need to  talk about
> the case where the NIC is handling many Sessions.  That is what tends to
> show the need for something to limit the impact on the NIC cost while
> permitting the Multi session 1-10 Gigabit links to operate at high speed
> and at long distances.

I have been making that very point.  The limited flow control features of
the iSCSI proposal necessitate connections per virtualized target ensuring
no single connection will share characteristics of a large pipe.  A relative
few segments of data processed ahead of recovery on any one connection will
have but a small impact overall.  A connection will cut throughput as a
result of loss to the benefit of other connections.

The cost trade-off is in respect to cost of holding the data obtained
following a loss before recovery.  This has the following components:
 1) The number of connections.
 2) The size of data, if a record mark is missed.
 3) The ratio of data to commands or status.
 4) Whether SACK is used.

As TCP shares bandwidth, if there are large numbers of connections, then how
any single connection is handled in an exceptional condition will have
minimal impact which should be true for iSCSI.  The NIC cost trade-off for
not using SACK and thus discarding past the loss will not greatly impact
maximum throughput obtained as again this is but a small fraction of overall
traffic.  iSCSI does not make recommendations as to the size of a data
transfer or limit these in a deterministic manner.  Even an occasional
record marking scheme will not ensure enough memory at an adapter to handle
every case of loss causing multiple scenarios of handling each case.  Status
and commands must be held and this aspect is difficult to assess as there is
yet no defined API for this type of intelligent adapter.  Whether the
adapter holds these items or has a means of signaling when they are valid,
this too would impact memory constraints.

> Second, in a many session environmentsb, if there is recovery being done
> for one session -- in cases where a high speed link is used that is at a
> significant distance -- then the impact on that one session will greatly
> effect others.  This is because the NIC must queue the partial data, while
> it is being recovered, and this can be a lot of data.

This is not likely given limited ability of iSCSI to regulate.  There is but
one immediate data allowed and a target wide command window.

> Third,assuming that a Urgent Pointer can be found, it means that, only the
> data before the Urgent Pointer needs to be queued on the NIC, and
> processing can continue with the other work, without reducing the TCP/IP
> window.

This question would need to consider all factors.  Problems in the use of
this urgent pointer is yet to be discovered.

Page 48 of the TCP spec (RFC 793) may provide a bit of an implementation
problem-

 "Note that data following the urgent pointer (non-urgent data) cannot
  be delivered to the user in the same buffer with preceding urgent data
  unless the boundary is clearly marked for the user."

Whether the urgent pointer is accurate, does not cause modified
acknowledgement behaviors or reduced performance with TCP accelerators makes
such use fraught with hazards.

> Fourth, I do not think that the implementations envisioned will
> be dropping
> the Data, instead it will  be Queueing it.  And so it will not be making
> the congestion any worse in the network.  The sooner the Urgent Ptr can be
> found, the less memory needed to hold the data on the card (while recovery
> for missing data is being done).  This means that there will be a lower
> probability that a slowdown will be needed for all the Sessions.

The TCP connection will slow as a result of loss.  The overall bandwidth
will not see any significant impact as there will not be any fat connections
to be of concern.

> If what I have stated in the four point above is true, then the need, and
> the solution, seem to  be a compelling match.  Matt, please check
> the above
> and let me know if I got it right.
>
> Now to the point about Must, Should, Will, might.....  I think the word
> Should is approprate.  I think we have Three groups of folks to worry
> about. Group L , Group E, and Group T.  Group L is the Low end (but
> probably high volume) Desktop and Laptop systems that will be using first
> 10/100 and over time 1000baseT connections.  Many of these
> systems will use
> normal SW TCP/IP stacks and will not be able to take advantage of the
> Urgent Pointer.  I do not think the extra overhead will be
> missed, at all,
> in this environment.  On the other hand, the E group (the Enterprise Class
> Servers) will need the fastest cards possible.    Likewise the T Group
> (targets),   will need to have as little overhead as possible, yet handle
> requests at the very highest speed level.  All  of these Groups (L , E, &
> T) need to have their requirements met, in order for iSCSI to be
> successful.  And if we think that iSCSI might have a chance to be used
> instead of FC, you must be concerned with the E & T Groups being happy.
>
> Therefore, every implementation SHOULD use the Urgent pointer.  Everyone
> wins when this works, even the Desktops, because even their 10/100
> connections, get trunked together with others into a very high band fiber
> that needs to enter the Target and  be serviced with very high speed.
> Hence the HW and its optimization and lowest approprate cost is KEY to
> everyone.

Trunking does not support an argument for Urgent Pointers.  Preventing the
use of TCP accelerators will impact all groups.  Dream or nightmare?  The
use of SHOULD to replace WILL in the proposal modifies TCP behavior.  In
reality, even if the Urgent Pointer option is indicated as a SHOULD, the
placement of the Urgent Pointer remains a MAY unless you wish to change TCP.
These problems associated with this atypical use of TCP prevents such a
requirement.  As usual, I do not share your vision.

Doug

> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> (408) 256-0403, Tie: 276-0403
> Internet address: hufferd@us.ibm.com
>
>