RE: iSCSI Autosense Consensus, Connection next steps

[julian_satran@il.ibm.com]

Joshua,

I agree that asymmetric is simpler and I personally would prefer it - but
playing devil's advocate (!)
- the simple backpresure mechanism and TCP windows and send buffers
proportional to the pipe rate will get you the skew down to acceptable
levels. The total buffer set needed is still proportional
to the total peak rate. And you will need similar mechanisms to regulate an
asymmetric flow too
(a bit simpler though!).

I am strongly in favor of the asymmetric model but I would like to hear all
possible arguments against it.

Julo

Joshua Tseng <jtseng@NishanSystems.com> on 06/09/2000 04:52:54

Please respond to Joshua Tseng <jtseng@NishanSystems.com>

To:   John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng
      <jtseng@NishanSystems.com>
cc:   ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM)
Subject:  RE: iSCSI Autosense Consensus, Connection next steps




John,

>My question then is: does Data Balancing on multiple
>connections cause a similar need for additional memory as does the
>synchronous connections with Multiple Commands on different connections.

In the symmetric model, the target must be prepared to buffer all of the
commands and associated unsolicited data PDU's that may be sent between
ExpCmdRN and MaxCmdRN.  This can be a lot of memory.  Either that or
discard
commands.  But in order to realize the performance benefits of command
windowing, the target must expand the window in order to let the initiator
to "let her rip!" and put multiple commands in flight.  It seems to me that
the existing symmetric model adds complexity which would only be of benefit
if the target has lots of memory.  So my question is this:  The command
windowing mechanism seems to require lots of resources (including memory)
and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to
implement, but does all this really buy very much?  In the end, symmetric
implementations may end up discarding commands anyway, as in the asymmetric
case, unless they want to install globs of memory.

WRT the load balancing algorithm implemented by iSCSI, anything other than
round robin will require more complexity to sense which connection is
over-utilized or poorly performing.  Once again, what does the added
complexity buy us?  I'm not an development engineer, but my guess is
getting
iSCSI to talk to TCP to get this info and use it isn't cheap.  In
asymmetric, commands are sent in-order down a relatively less-utilized
command TCP connection.  A slow TCP connection only affects unfortunate
exchanges whose data PDU's happen to use that connection, and not all
sessions between initiator and target.

Josh



-----Original Message-----
From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com]
Sent: Tuesday, September 05, 2000 4:53 PM
To: Joshua Tseng
Cc: ips@ece.cmu.edu
Subject: RE: iSCSI Autosense Consensus, Connection next steps



Joshua,
I think some of your points are correct.  But we also need to examine a few
statements.

First, the round robin approach of balancing, if only done in the manor you
specified,  would of course cause every thing to slow to the speed of the
slowest link.  But this is a well known problem, and general managed by a
modified round robin so that it continues to other connections if one of
the connections is still busy (several different way of determining this).
In fact this would be a problem in the current wedge drivers today used
with Fibre Channels if only a blind round robin approach was used.

Your point about the target perhaps needing additional memory, is probably
correct.  This is also solvable on the initiator side (by a number of
techniques)  however, the target can not be sure that the initiator will
have the right techniques, so it must assume that it needs additional
memory.  This seems like a correct point.  If this is a worry, however, the
Target can just reject additional conversations per Session.  The counter I
would make to my last point, however, would be if Asynchronous was used, I
could still could handle multiple data ports and at least get that kind of
balancing.  My question then is: does Data Balancing on multiple
connections cause a similar need for additional memory as does the
synchronous connections with Multiple Commands on different connections.

On the other hand, think that your point about link recovery not being
(significantly) more difficult with Async then Sync is essentially correct.

.
.
.
John L. Hufferd



Joshua Tseng <jtseng@NishanSystems.com>@ece.cmu.edu on 09/05/2000 02:54:56
PM

Sent by:  owner-ips@ece.cmu.edu


To:   Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu
cc:
Subject:  RE: iSCSI Autosense Consensus, Connection next steps



David,

If I may give my input on the TCP connection model discussion, I'd like
to advocate the asymmetric model for the following reasons.  Using
multiple TCP connections, each TCP connection will likely have
different performance characteristics.  If this is so, then some commands
may reach the target out of order from other commands sent down different
TCP connections.  Commands sent down the faster connection will then need
to be buffered until commands sent down slower connections arrive.  This
will lead to the following problems that I can think of:

1)  There will be increased memory requirements for the iSCSI target,
which must buffer commands and related data and reorder them before passing
them on to the SCSI controller.

2)  If the initiator cannot sense the performance characteristics of
each TCP connection, then interactions between initiator and target will
be dependent on the slowest TCP connection.  For example, if there are
five TCP connections and one of them is extraordinarily slow, and the
initiator merely round-robins the commands down each of the five
connections,
all interactions between initiator and target will be only as fast as the
slowest connection.

In addition, it has been stated that detecting and recovering from a failed
asymmetric control connection is more complicated and difficult than
detecting and recovering from a failed symmetric data/command TCP
connection.
I do not understand why this is so.  In both cases, iSCSI initiator will
need
to abort the lost command AND those following the lost command, and resend
all
commands in a new or different TCP connection.  In the symmetric case,
iSCSI
commands received by a target following a lost command will not be
acknowledged,
since ExpStatRN is a cummulative windowing mechanism.

Josh Tseng

-----Original Message-----
From: Black_David@emc.com [mailto:Black_David@emc.com]
Sent: Sunday, September 03, 2000 8:20 AM
To: julian_satran@il.ibm.com; ips@ece.cmu.edu
Cc: Black_David@emc.com
Subject: iSCSI Autosense Consensus, Connection next steps


> It looks like we have consensus - but the chairmen have the call.

Indeed we do, and I apologize for the delay, due to an inability to
get connected.

First, on Autosense, I believe that rough consensus exists for
iSCSI to require Autosense.  I have seen only one objection;
If anyone other than Doug Otis disagrees with this, please
say so on the list.

Second, on connections, I haven't seen enough discussion to call
consensus, but I am going to try to narrow the option space and
structure the discussion.  Four models for sessions have been
proposed:

(1) Symmetric - all connections usable for command and data.
(2) Asymmetric - single command connection, others are data.
(3) Split - assign LUN sets to specific connections or pools of
     connections.
(4) SCTP - use SCTP's support for multiple connections.

I have only seen one message suggesting/supporting (3) Split, so
I believe rough consensus exists to not pursue that model
any further.  If anyone other than Paul von Stamwitz disagrees,
and believes that the Split model should be pursued, please
say so on the list with technical rationale.

I haven't seen enough discussion on use of SCTP instead of
TCP to call consensus.  I am concerned that this list is not
going to produce a consensus on the issue phrased in that
fashion (i.e., pick exactly one).  The one point on which there
does seem to be consensus is that SCTP is a considerably
younger protocol that TCP, and hence the likely timelines
to availability of hardware acceleration seem to favor TCP.
OTOH, this sort of future prediction can easily miss the mark.

I can see two possible ways to make progress in this area:
- An off-line design team to do an intensive evaluation of SCTP
     vs. TCP for iSCSI.
- Recognize the merits of SCTP as well as TCP, plan for both
     with the anticipation that TCP will be used first.
The latter makes more sense to me, because it appears to
be more amenable to consensus, avoids the investment
of effort in the design team, and avoids having the prospect
of abandoning TCP hanging over the heads of those engaged
in TCP-specific work.  The practical implication of proceeding
in this fashion is that SCTP-friendliness becomes an additional
criteria to use in evaluating iSCSI design proposals.  Proceeding
in this fashion is only a proposal at this juncture -- please
comment on whether this is the right way to proceed (to me
or on the list).

The specification of sessions for iSCSI over TCP needs to
proceed, so under the assumption that either the "plan for
both" path will be pursued, or that we can't wait for the
design team's conclusions, and hence need to work on TCP
in case it is selected, I want to summarize the tradeoffs
between the Symmetric and Asymmetric session models,
in the aim of simulating more discussions so that we can
get to consensus.

[X] The Symmetric model imposes additional work on implementations
     that do not use multiple connections because they will
     have to implement the command numbering.  The
     Asymmetric model does not do this.
[Y] The Asymmetric model makes dealing with a failed control
     connection more complicated than the symmetric model
     because agreement between the two sides is required
     to establish a new control connection (or use an existing
     data connection for control).

I have seen the discussion of [X] favoring Asymmetric for systems
that will not support multiple TCP connections per iSCSI session.
I also note (as an example of applying the SCTP-friendliness
from above) that [X] favoring Asymmetric also applies to SCTP, as
there seems to be no point in using multiple SCTP sessions in a
single iSCSI session.

We need to give the folks working on revising the iSCSI draft some
direction on this issue in fairly short order, so comments are hereby
solicited ...

Thanks,
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909
black_david@emc.com  Cellular: +1 (978) 394-7754
---------------------------------------------------