Re: Avoiding deadlock in iSCSI

[julian_satran@il.ibm.com]

Costa,

The deadlock - that we keep coming back to - can be avoided in the
asymmetric case
by a set of simple rules.

On any GIVEN connection send the data in the same order you sent the
commands
(and I mean ALL the data from a command).

This combined with the fact that data arrive on known connections (Kalman's
proposal)
will enable a target to accept data when executing commands and not having
to keep
around data when it doesn't need it.

For the symmetric case follow the same rule but ADD THE FOLLOWING:

if on a connection you have read a command that is out of order stop
reading from it
until the gap is filled (I assume that on any GIVEN connection commands
have non-decreasing numbers)

If you intend to use execution orders of the more exotic kind then never
use
unsolicited data.

Again we are not far better of with neither asymmetric or symmetric - but
the
asymmetric is simpler.

And if you have a bad initiator - the only worry we should have is that he
should not harm others until we can fence him off.

On the asymmetric case you could start fencing if the initiator is sending
data on any
given connection out of order; in the symmetric case if it sends commands
or data
out of order. Again not that different.

Julo

csapuntz@cisco.com on 12/09/2000 01:02:42

Please respond to csapuntz@cisco.com

To:   ips@ece.cmu.edu
cc:   csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM)
Subject:  Avoiding deadlock in iSCSI





The problem:

iSCSI, as currently spec'ed, allows SCSI commands and data to be
interleaved fairly freely on a TCP connection. A target that stops
reading from a TCP connection to avoid reading more command packets
also prevents itself from reading data packets.  Those data packets
may be criticial to making progress on the currently executing
command.

Note the issue appears with one TCP connection for control and data
and even appears in many of the multiple connection schemes.

Data in iSCSI comes in two forms:

     1) solicited - data requested by target via RTT
                  - data requested by initiator via a SCSI command
     2) unsolicited - data sent by initiator without having received an RTT

The analysis below assumes that unsolicited data travels over the same
TCP connection as SCSI commands. Otherwise, you run the risk of receiving
unsolicited data before the relevant SCSI command (thus making
implementations more complex).

Four solutions:

1) Don't overflow the command queue (i.e. use credits)
     - and what do you do if a misbehaving initiator overflows
        your command queue anyway? Drop the connection?

     - requires you to reserve resources per initiator. some people
        may want to overcommit

2) Allow dropping of SCSI commands when queue fills
     - how do you clean up after a dropped SCSI command?
         - there may be other commands in the pipeline

     One approach: On command drop, the target enters an error
     state. While in the error state, all newly received commands
     terminate with an error until the initiator explicitly clears
     the error state using a "clear error state" message.

     You might think that TASK SET FULL and ACA mechanisms from SCSI
        could be used to attack this problem. However, TASK SET FULL errors
     don't trigger ACA (in my reading of the SAM). Also, ACA is only
     triggered by the current enabled command, not by random commands
     entered into the task set.

3) Put solicited data on a dedicated TCP connection. Require that
unsolicited data MUST follow the command, ideally in the same iSCSI
PDU

4) (Do it like NFS) Make all transfers from initiator to target
unsolicited. Make sure unsolicited data follows the command
immediately.


Of all the options, #1 and #4 sound the easiest to implement. #2 is more
sophisticated than #1. #3 is just plain clever but that's rarely a good
thing. :)  #4 has large ramifications on current SCSI target designs.

-Costa