RE: Avoiding deadlock in iSCSI

["Y P Cheng" <ycheng@advansys.com>]

Costa wrote:
>iSCSI, as currently spec'ed, allows SCSI commands and data to be
>interleaved fairly freely on a TCP connection. A target that stops
>reading from a TCP connection to avoid reading more command packets
>also prevents itself from reading data packets.  Those data packets
>may be criticial to making progress on the currently executing
>command.
>
>Note the issue appears with one TCP connection for control and data
>and even appears in many of the multiple connection schemes.
>
>Data in iSCSI comes in two forms:
>	1) solicited - data requested by target via RTT
>             - data requested by initiator via a SCSI command
>	2) unsolicited - data sent by initiator without having received an RTT

I appreciate your examples.
On incoming SCSI commands, the SCSI-QUEUE-FULL status is used to tell an
initiator that it needs to resend the command again.  Most initiator avoids
this status by tracking how many commands being sent to each SCSI device.
On unsolicited data, it is tough because a target does not have unlimited
resources with so many potential initiators on the net.  RTT can force the
data being solicited.  Another method is to use the queue-depth parameters
to each initiators and taking a chance on over-subscribing.  However, when
an incoming PDU is dropped, the target must inform the initiator.  This can
be done in the microcode of a NIC adapter easily but not necessarily in an
IP device driver.  In our NIC implementation, all incoming data must be
solicited.  Even so, some PDUs can still be dropped because of slow host
system bus.  No error goes unreported.  Even so, the initiator still has the
responsibility of detecting if PDU's are dropped due to traffic jam on the
gateway or due to broken connections.

Y.P. Cheng, CTO, ConnectCom Solutions Corp.

-----Original Message-----
From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
csapuntz@cisco.com
Sent: Monday, September 11, 2000 3:03 PM
To: ips@ece.cmu.edu
Cc: csapuntz@cisco.com
Subject: Avoiding deadlock in iSCSI



The problem:

iSCSI, as currently spec'ed, allows SCSI commands and data to be
interleaved fairly freely on a TCP connection. A target that stops
reading from a TCP connection to avoid reading more command packets
also prevents itself from reading data packets.  Those data packets
may be criticial to making progress on the currently executing
command.

Note the issue appears with one TCP connection for control and data
and even appears in many of the multiple connection schemes.

Data in iSCSI comes in two forms:

	1) solicited - data requested by target via RTT
	             - data requested by initiator via a SCSI command
	2) unsolicited - data sent by initiator without having received an RTT

The analysis below assumes that unsolicited data travels over the same
TCP connection as SCSI commands. Otherwise, you run the risk of receiving
unsolicited data before the relevant SCSI command (thus making
implementations more complex).

Four solutions:

1) Don't overflow the command queue (i.e. use credits)
	- and what do you do if a misbehaving initiator overflows
        your command queue anyway? Drop the connection?

	- requires you to reserve resources per initiator. some people
        may want to overcommit

2) Allow dropping of SCSI commands when queue fills
	- how do you clean up after a dropped SCSI command?
	    - there may be other commands in the pipeline

	One approach: On command drop, the target enters an error
	state. While in the error state, all newly received commands
	terminate with an error until the initiator explicitly clears
	the error state using a "clear error state" message.

	You might think that TASK SET FULL and ACA mechanisms from SCSI
        could be used to attack this problem. However, TASK SET FULL errors
	don't trigger ACA (in my reading of the SAM). Also, ACA is only
	triggered by the current enabled command, not by random commands
	entered into the task set.

3) Put solicited data on a dedicated TCP connection. Require that
unsolicited data MUST follow the command, ideally in the same iSCSI
PDU

4) (Do it like NFS) Make all transfers from initiator to target
unsolicited. Make sure unsolicited data follows the command
immediately.


Of all the options, #1 and #4 sound the easiest to implement. #2 is more
sophisticated than #1. #3 is just plain clever but that's rarely a good
thing. :)  #4 has large ramifications on current SCSI target designs.

-Costa