RE: Avoiding deadlock in iSCSI

["Douglas Otis" <dotis@sanlight.net>]

John,

With separate TCP connections, there is no guarantee which connection will
be sending with the lowest latency.  One would expect a command stream to be
less, but it may be without inertia.  A command given to the NIC ahead of
data may arrive after the data.  The arrival time of two streams sending
different quantities would be difficult to predict.  As such, out of
sequence delivery.  Would you recommend a skew buffer?

Doug

> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> John Hufferd/San Jose/IBM
> Sent: Monday, September 11, 2000 11:50 PM
> To: ips@ece.cmu.edu
> Subject: RE: Avoiding deadlock in iSCSI
>
>
> Costa,
> I think I understand your points, but please also consider the Asymmetric
> approach.  Please consider the Asymmetric approach and also as I
> said in my
> last note,  suppose that we "make a rule, that no unsolicited data can be
> sent, before its command is sent AND that unsolicited data must be sent in
> the same order as the commands that reference it.  Along with that rule,
> and the statements below from David -- will, in your opinion, the
> Asymmetric approach work even with unsolicited data?
>
> .
> .
> .
> John L. Hufferd
>
> ---------------------- Forwarded by John Hufferd/San Jose/IBM on
> 09/11/2000
> 11:19 PM ---------------------------
>
> David Robinson <David.Robinson@EBay.Sun.COM>@ece.cmu.edu on 09/11/2000
> 06:52:22 PM
>
> Please respond to David Robinson <David.Robinson@EBay.Sun.COM>
>
> Sent by:  owner-ips@ece.cmu.edu
>
>
> To:   ips@ece.cmu.edu
> cc:
> Subject:  RE: Avoiding deadlock in iSCSI
>
>
>
> Thanks for your clarifying comments.
>
> > > In general
> > > I consider that to be a bug and the receiver should just drop the
> > > data on the floor.
> >
> > Not as I understand solicit.
>
> This I don't understand, it you get data that no command has been
> sent yet, as the reciever you can either wait and hope the
> command arrives consuming buffer space but eventually dropping
> it after some timeout, or drop it on the spot. In either case this
> seems like a bad design we should try to avoid.
>
> > > My first assumption is that the sender would not send commands
> > > C1 and C2 and data D2 and D1 on the same connection. Doing that
> > > creates nasty ordering problems we want to avoid.
> >
> > Order on the wire can not be controlled. Only the ULP can avoid such.
>
> Yes it can, this is exactly the advantage of using a reliable stream
> protocol like TCP, the session layer never sees out of order packets.
> With multiple data connections and the appropriate ordering constraints
> we have no deadlock or buffer management issues.
>
> > Resources are held until associated data is received to complete
> operations.
> > If the resource limit is not the data buffer nor freed by
> content already
> > within the data buffer, this will result in discarding commands.
>
> But with a reliable stream no commands need to be discarded, the
> transport flow controls so the commands are held at the sender.
>
> > > With multiple data connections, some may flow
> > > control but the active command will be able to make progress on
> > > one connection. This may not be the most efficient mechanism but
> > > it is "safe".
> >
> > One connection per LUN or one connection per command, safe but
> expensive?
>
> Define "expensive", not in terms of performance as one TCP
> connection can saturate the link layer or not in terms of memory
> as the mux/demux state has to be held either in the transport or
> the session layer.  I am not advocating a connection per command
> as that is just a bad datagram protocol, but either a connection per
> LUN or per target should work just fine.
>
> > As a means for freeing resources, data is to be discarded within the
> iSCSI
> > architecture.  As such, even unsolicited data may be requested by the
> > target.
>
> I don't understand this statement. Short of target errors or connection
> errors, why does data need to be discarded? The sender should never
> send data without a command, and on a given connection the data MUST
> always be sent after the corresponding command and data from two
> commands must always be sent in order if on the same connection.
>
>      -David
>
>
>
>