Re: iSCSI - Change Proposal X bit

["Julian Satran" <Julian_Satran@il.ibm.com>]

Santosh,

Some comments in text.  I suggest you discuss the scenarios with 
Mallikarjun or if you can't do that for some reason
please call me. We can't go on with this ad nauseam on the list.

Julo




Santosh Rao <santoshr@cup.hp.com>
Sent by: owner-ips@ece.cmu.edu
24-10-01 18:29
Please respond to Santosh Rao

 
        To:     Julian Satran/Haifa/IBM@IBMIL
        cc:     ips@ece.cmu.edu
        Subject:        Re: iSCSI - Change Proposal X bit

 

Julian,

Some comments on the below quoted scenarios :

>    session has 3 connections
>    on connection 1 I->T c1,c2,c3,C6
>    on connection 2 I->T c4,c5,c7,c8
>    Target receives 1,2,4,5,7,8 (miss 3 and 6) and acks 1 & 2
>    Initiator closes 1 and resends c3, c4, c5,c7,c8  on connection 2 and 
6
>    on connection 3
>    target receives all and starts executing and acks 8 on connection 3 
but
>    connection 2 stalls after c3 for a  LONG TIME
>    then (after 2 full sequence wraps) connection 2 is gets alive and
>    delivers c4,c5 etc (that are now valid)

When the target acks CmdSN 8 on connection 3, it has, in effect, sent
CmdSN ack's for CMdSNs 3,4,5,6,7,8. This implies that the commands with
CmdSN 3, 4, 5, 7, & 8 were received by the target on connection 2 and
their processing was commenced.

+++ to understand fully the scenario make a timeline and observe taht 
after 3 and 6 where received 4-8 can be executed and their answers sent 
(connection 2 has not died). But c3 and the rest reach the pipe (depending 
on implementation this may happen after statuses have reached the 
initiator and TCP acks have been sent) and then the pipe stalls with the 
duplicates in the pipe without the target knowing about them+++

Hence, the following does not make sense :

>    connection 2 stalls after c3 for a  LONG TIME
>    then (after 2 full sequence wraps) connection 2 is gets alive and
>    delivers c4,c5 etc (that are now valid)

c4, c5, etc were already delivered to the target and are not being
re-delivered. There is no problem in this case. (??).

Take the next scenario :
> 2 connections:
> 
> connection1  I->T c3,c4,c5
>       status of 3 contains ack up to 6 and it and all other statuses are
> lost
> connection2  resend c3, c4 & c5 (no logout) and those are executed!

Since the initiator got CmdSN ack's upto 6, the initiator should not be
re-issuing these I/Os ??

I still don't see justification to require that initiators send a
immediate NOP-OUT in the manner being advocated.


On a more fundamental note, I see some issues with the initiator being
allowed to re-issue the commands on a different connection without
having first logged out the previous connection successfully. I see
nothing in the draft that suggests such behaviour, while at the same
time, it is not forbidden.

+++ the problem appears WHEN THE INITIATOR REISSUES THE COMANDS ON THE 
SAME CONNECTION +++

By resorting to command retries on a different connection in an attempt
to plug the hole, without first logging out the previous connection, the
initiator is susceptible to encountering I/O failure of that I/O due to
ULP timeout.

+++ That is not what the first scenario says. The second is trickier and 
involves the fact

Here's the scenario why such recovery should not be allowed :
- Initiator sends CmdSN 3 on connection 1.
- No CmdSN updates for a while and initiator re-sends CmdSn 3 on
connection 2.
- At the same time, target has sent CmdSN ack's for CmdSN 3 on
connection 1.

- Initiator has transferred the command allegiance on its side from
connection 1 to connection 2 and is attempting the command on connection
2. However, the command does not go through, since the (ExpCmdSN,
MaxCmdSN) window has advanced and the trget discards the command.

- Target sends in data and/or R2T and/or status for CmdSN 3 on
connection 1. Since the initiator is not expecting any traffic for that
I/O on connection 1, it discards any PDUs received on that connection 1
for which no I/O state existed.

In the above scenario, initiator will never get a CmdSN ack on
connection 2 and will never be able to plug the hole despite repeated
retries, finally, causing a ULP timeout, followed by session recovery.

Given the above scenario, I suggest that the initiator must only
re-issue commands on the same connection, and can re-issue them on
another connection only following a successful logout.

+++ That is what the current text says explicitly.  Scenario 2 has not 
been properly stated. +++
Comments ?

Thanks,
Santosh


Julian Satran wrote:
> 
> Santosh,
> 
> The scenarios I am talking about are all derivatives of an initiator 
trying
> to plug-in holes and switching connections.
> As the initiator does know the "extent" of a hole it can send-out 
commands
> that he did not have to.
> I have sent the attached not to Mallikarjun  a while ago.  I think that
> there might be many of this kind.  I am also aware that X bit by itself
> might have some bad scenarios but the new proposal fixes them all.
> 
> Julo
> 
> _____________________________
> 
> Mallikarjun,
> 
> Take the following sequence scenario:
> 
>    session has 3 connections
>    on connection 1 I->T c1,c2,c3,C6
>    on connection 2 I->T c4,c5,c7,c8
>    Target receives 1,2,4,5,7,8 (miss 3 and 6) and acks 1 & 2
>    Initiator closes 1 and resends c3, c4, c5,c7,c8  on connection 2 and 
6
>    on connection 3
>    target receives all and starts executing and acks 8 on connection 3 
but
>    connection 2 stalls after c3 for a  LONG TIME
>    then (after 2 full sequence wraps) connection 2 is gets alive and
>    delivers c4,c5 etc (that are now valid)
> 
> That is not a very likely scenario, I admit, but it is possible.
> With X bit I could not find any such scenario since an X either follows 
a
> good one on the same connection or can be safely discarded.
> I suspect that there are some more scenarios that involve immediate
> commands or commands that carry their own ack in the status and are 
acked
> like:
> 
> 2 connections:
> 
> connection1  I->T c3,c4,c5
>       status of 3 contains ack up to 6 and it and all other statuses are
> lost
> connection2  resend c3, c4 & c5 (no logout) and those are executed!
> 
> I think we can avoid those be requiring a NOP exchange before reissuing 
a
> command on a new connection or reissue the command with a task 
management
> (that has an implied ordering) but why do it if X is an obvious and safe
> solution.
> 
> Julo
> 
> Regards,
> Julo
> 
> 
>                     "Mallikarjun
>                     C."                  To:     Julian 
Satran/Haifa/IBM@IBMIL
>                     <cbm@rose.hp.c       cc:
>                     om>                  Subject:     Re: iscsi : X bit 
in SCSI Command PDU.
> 
>                     08-10-01 21:45
>                     Please respond
>                     to cbm
> 
> 
> 
> Julian,
> 
> We currently have the following specified in section 2.2.2.1 -
> 
> "The target MUST NOT transmit a MaxCmdSN that is more than
> 2**31 - 1 above the last ExpCmdSN."
> 
> It appears to me that the above is sufficient to ward off the
> accidents of the sort you describe.  Do you think otherwise?
> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> MS 5668 Hewlett-Packard, Roseville.
> cbm@rose.hp.com
> 
> Julian Satran wrote:
> >
> > Mallikarjun,
> >
> > There is at least one theoretical scenario in which an "old" command
> > may appear in a "new window" and be reinstantiated.
> > At 10Gbs and several connection that does not take months. With X the
> > probability is far lower (not 0).   I have no other strong arguments
> > but I am still  thinking.  Matt Wakeley that insisted on it (against
> > me) had some other argument that I am trying to find (I am note
> > remembering).
> >
> > Julo
> >
> >   "Mallikarjun C."
> >   <cbm@rose.hp.com>                  To:        Julian
> >                              Satran/Haifa/IBM@IBMIL
> >   08-10-01 20:39                     cc:
> >   Please respond to cbm              Subject:        Re: iscsi : X
> >                              bit in SCSI Command PDU.
> >
> >
> >
> > Julian,
> >
> > Now that you put me on the spot, :-), my response -
> >
> > Santosh argued with me privately that X-bit no longer serves a
> > useful purpose after the advent of task management commands to
> > reassign.  My response was that it never was a requirement per se,
> > but always a "courtesy" extended by the initiator to help the
> > target.  I also suggested that X-bit may be considered for its
> > usefulness in debugging.
> >
> > He still had some (very reasonable) comments for simplification
> > - the most appealing of which (to me) was the opportunity to do
> > away with the X-bit checking for *every* command PDU that the target
> > has to endure now.
> >
> > If I missed a legitimate use of X-bit, please comment. Do you
> > think it is a protocol requirement per se?  I couldn't justify
> > to myself so far (except the Login).
> >
> > Regards.
> > --
> > Mallikarjun
> >
> > Mallikarjun Chadalapaka
> > Networked Storage Architecture
> > Network Storage Solutions Organization
> > MS 5668 Hewlett-Packard, Roseville.
> > cbm@rose.hp.com
> >
> >
> >
> > Julian Satran wrote:
> > >
> > > Santosh,
> > >
> > > I am not sure you went through all scenarios. A conversation with
> > your
> > > colleague - Mallikarjun - and getting through the state table may go
> > a
> > > long way to clarify the need for X.
> > >
> > > And I am sure that by now you found yourself several .
> > >
> > > Julo
> > >
> > >   Santosh Rao
> > >   <santoshr@cup.hp.com>                   To:        IPS Reflector
> > >   Sent by: owner-ips@ece.cmu.edu  <ips@ece.cmu.edu>
> > >                                           cc:
> > >   06-10-01 01:56                          Subject:        iscsi : X
> > >   Please respond to Santosh Rao   bit in SCSI Command PDU.
> > >
> > >
> > >
> > > All,
> > >
> > > With the elimination of command relay from iscsi [in the interests
> > of
> > > simplification ?], I believe that the X bit in the SCSI Command PDU
> > > can
> > > also be removed. As it exists today, the X bit is only being used
> > for
> > > command restart, which is at attempt by the initiator to plug a
> > > potential hole in the CmdSN sequence at the target. It does this on
> > > failing to get an ExpCmdSN ack for a previously sent command within
> > > some
> > > timeout period.
> > >
> > > Given the above usage of command restart, no X bit is required to be
> > > set
> > > in the SCSI Command PDU when command re-start is done.
> > >
> > > Either :
> > > (a) the target had dropped the command earlier due to a digest
> > error,
> > > in
> > > which case, the command restart plugs the CmdSN hole in the target.
> > >
> > > [OR]
> > >
> > > (b) the target had received the command and was working on it, when
> > > the
> > > initiator timed out too soon and attempted a command restart to plug
> > > [what it thought was] a possible hole in the CmdSN sequence.
> > >
> > > In case (a), no X bit was required, since the target knows nothing
> > of
> > > the original command. In case (b), no X bit is required again, since
> > > the
> > > (ExpCmdSN, MaxCmdSN) window would have advanced and the target can
> > > silently discard the received retry and continue working on the
> > > original
> > > command received.
> > >
> > > Removal of the X bit in the SCSI Command PDU has the following
> > > benefits
> > > :
> > >
> > > a) The CmdSN rules at the target are simplified. No need to look at
> > X
> > > bit, only validate received CmdSN with (ExpCmdSN, MaxCmdSN) window.
> > >
> > > b) The reject reason code "command already in progress" can be
> > > removed.
> > > There's no need for this reject reason code anymore, since X bit
> > > itself
> > > is not required, and the targets can silently discard commands
> > outside
> > > the command window and continue to work on the original instance of
> > > the
> > > command already being processed at the target.
> > >
> > > c) Less work for the target and less resources consumed since it no
> > > longer needs to generate a Reject PDU of type "command in progress".
> > > It
> > > can just silently discard any command PDU outside the (ExpCmdSN,
> > > MaxCmdSN) window.
> > >
> > > d) Less code for the target, since it does not need :
> > > - any Reject code paths when it receives X bit command PDUs that are
> > > already in progress.
> > > - No special casing of CmdSN checking rules.
> > > - No overheads of verifying a received command based on its
> > initiator
> > > task tag, to check if the task is currently active, prior to sending
> > a
> > > Reject response with "command in progress".
> > >
> > > Comments ?
> > >
> > > Thanks,
> > > Santosh
> > >
> > > --
> > > ##################################
> > > Santosh Rao
> > > Software Design Engineer,
> > > HP-UX iSCSI Driver Team,
> > > Hewlett Packard, Cupertino.
> > > email : santoshr@cup.hp.com
> > > Phone : 408-447-3751
> > > ##################################
> 
> 
>                     Santosh Rao
>                     <santoshr@cup.       To:     IPS Reflector 
<ips@ece.cmu.edu>
>                     hp.com>              cc:
>                     Sent by:             Subject:     Re: iSCSI - Change 
Proposal X bit
>                     owner-ips@ece.
>                     cmu.edu
> 
> 
>                     23-10-01 22:50
>                     Please respond
>                     to Santosh Rao
> 
> 
> 
> Julian Satran wrote:
> >
> > However in order to drop "old" commands that might in the pipe on a
> > sluggish connection - removing the X bit will require the initiator to
> > issue an immediate NOP requiring a NOP response on every open 
connection
> > whenever CmdSN wraps around (becomes equal to InitCmdSN).
> 
> Julian,
> 
> Can you please explain further the corner case you are describing above
> ? Are you suggesting that special action should be taken every time
> CmdSN wraps around, in case there were holes in the CmdSN sequence at
> the wrap time ? Why is that ?
> 
> Here's my understanding of how this plays out :
> 
> Rule 1)
> The CmdSN management rules at the target should be handling CmdSN wrap
> case and the initiator cannot issue more than 2^32 -1 commands beyond
> the last ExpCmdSN update it has received from the target, since the
> target MUST NOT transmit a MaxCmdSN that is more than 2**31 - 1 above
> the last ExpCmdSN. (per Section 2.2.2.1)
> 
> Rule 2)
> Any holes that occur in the CmdSN sequence are attempted to be plugged
> by the initiator by re-issuing the original command. If the CmdSN never
> got acknowledged and the I/O's ULP timeout expired, the initiator MUST
> perform session recovery. (per Section 8.6)
> 
> Thus, going by the above 2 rules, if the CmdSN sequence wraps upto
> ExpCmdSN, the initiator will not be able to issue further commands,
> since the target will keep the CmdSN window closed. The window can only
> re-open when the CmdSN holes are plugged allowing ExpCmdSN and thereby,
> MaxCmdSN to advance.  (rule 1 above).
> 
> Under the above circumstances, the initiator will possibly try to plug
> the CmdSN hole by re-issuing the original command. It may do this 1 or
> more times before its ULP timeout expires. Either the holes get plugged
> and the windoe re-opens, or ULP timeout occurs without the corresponding
> CmdSN for that I/O having been acknowledged, resulting in session
> logout. (rule 2 above).
> 
> What is required over and beyond the above ? Why does removal of X-bit
> require an immediate NOP to be issued every time CmdSN wraps and a hole
> exists in the CmdSN sequence (??).
> 
> Regards,
> Santosh
> 
> --
> ##################################
> Santosh Rao
> Software Design Engineer,
> HP-UX iSCSI Driver Team,
> Hewlett Packard, Cupertino.
> email : santoshr@cup.hp.com
> Phone : 408-447-3751
> ##################################

-- 
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : santoshr@cup.hp.com
Phone : 408-447-3751
##################################