RE: iSCSI: Addition of CmdSN in Data-out PDU

["Julian Satran" <Julian_Satran@il.ibm.com>]

Robert,

The ordering rule is for a connection (not for a session).  It's sole
purpose is to minimize the chance for a deadlock.

Regards,
Julo


                                                                                        
                    Robert Snively                                                      
                    <rsnively@broc       To:     "'Rod Harrison'"                       
                    ade.com>              <rod.harrison@windriver.com>, Julian          
                                          Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu       
                    16-10-01 18:02       cc:                                            
                    Please respond       Subject:     RE: iSCSI:  Addition of CmdSN in  
                    to Robert             Data-out PDU                                  
                    Snively                                                             
                                                                                        
                                                                                        



I am not quite sure what meaning "in-order" data has in
a multi-connection session.  Does it mean that you actually
expect the data across multiple connections to arrive temporally
in the same order that is specified by the command sequence
numbers?  Or does it mean that the data has to be arranged
in some order by the target session processing?

Or is it better to step aside from the in-order question
for data transmission and do as Rob Harrison suggests:

"As an initiator I would send the command PDU as soon as
I could if there were no immediate data, then queue the
DMA for the unsolicited data. Since there could be many
other DMAs queued ahead of the unsolicited data it is quite
possible that other commands, both read and write, will
be sent before the DMA completes."

This would imply no strict ordering requirement.

If an ordering requirement is necessary, I submit that it
is because of the problems with unsolicited data reception,
not with the actual data order.

Bob

> -----Original Message-----
> From: Rod Harrison [mailto:rod.harrison@windriver.com]
> Sent: Monday, October 15, 2001 4:06 AM
> To: Julian Satran; ips@ece.cmu.edu
> Subject: RE: iSCSI: Addition of CmdSN in Data-out PDU
>
>
> Julian,
>
>          I think you might have misunderstood what I meant. I was not
> advocating we change the spec in favour of relaxing the data ordering
> requirements. I was just point out there is a problem for the
> initiator to solve in order maintain correct ordering, and that I
> believe it is not a terribly difficult one.
>
>          - Rod
>
> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Julian Satran
> Sent: Sunday, October 14, 2001 9:30 PM
> To: ips@ece.cmu.edu
> Subject: RE: Addition of CmdSN in Data-out PDU
>
>
>
> Immediate data are not mandatory (even if enabled). Having one or ten
> DMAs
> does not change the fact that they share a TCP "pipe" and a decent
> transmitter will ship the TCP data in order - otherwise it just throws
> his
> problems in the receiver's lap. Our additional requirement on order
> should
> make no difference.  And it would be time to change the name of the
> thread.
>
> Julo
>
>
>
>
>
>                     "Rod Harrison"
>                     <rod.harrison@wind       To:     John Hufferd/San
> Jose/IBM@IBMUS, "Robert
>                     river.com>                Snively"
> <rsnively@Brocade.COM>
>                     Sent by:                 cc:
> <somesh_gupta@silverbacksystems.com>,
>                     owner-ips@ece.cmu.        "BURBRIDGE,MATTHEW
> (HP-UnitedKingdom,ex2)"
>                     edu
> <matthew_burbridge@hp.com>, "'Binford, Charles'"
>                                               <CBinford@pirus.com>,
> <ips@ece.cmu.edu>
>                                              Subject:     RE: Addition
> of CmdSN in Data-out PDU
>                     13-10-01 13:12
>                     Please respond to
>                     "Rod Harrison"
>
>
>
>
>
>
>            Even more common would be the DMA latency. As an initiator
> I
> would
> send the command PDU as soon as I could if there were no immediate
> data, then queue the DMA for the unsolicited data. Since there could
> be many other DMAs queued ahead of the unsolicited data it is quite
> possible that other commands, both read and write, will be sent before
> the DMA completes.
>
>            You are spot one when you say the delay in obtaining the
> unsolicited
> data may be large and variable. If there are multiple DMA engines it
> does become a challenge for the initiator to ensure the correct
> ordering of the unsolicited data. Note that since there may be
> multiple unsolicited data PDUs in the same burst, and those may be
> filled with separate DMAs the ordering problem propagates down to
> within individual commands.
>
>            I don't believe it is a particularly difficult problem for
> the
> initiator to solve, but I do agree things would be easier for the
> initiator if the ordering requirement were removed. And more so if the
> requirement to send in dataSN order were also removed, but this is
> probably too much pain for the target.
>
>            - Rod
>
> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> John Hufferd
> Sent: Saturday, October 13, 2001 1:01 AM
> To: Robert Snively
> Cc: 'somesh_gupta@silverbacksystems.com'; BURBRIDGE,MATTHEW
> (HP-UnitedKingdom,ex2); 'Binford, Charles'; ips@ece.cmu.edu
> Subject: RE: Addition of CmdSN in Data-out PDU
>
>
>
> Robert,
> I think that an Initiator being able to send a waiting Read command,
> without having to wait for many large write segments -- that are being
> sent
> (as unsolicited data) -- is very useful.  And that would mean, the
> unsolicited data is waiting to be sent until the Read Commands are
> sent.
> This might be a very frequent case.
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136
> Internet address: hufferd@us.ibm.com
>
>
> Robert Snively <rsnively@Brocade.COM>@ece.cmu.edu on 10/12/2001
> 03:56:06 PM
>
> Sent by:  owner-ips@ece.cmu.edu
>
>
> To:   "'somesh_gupta@silverbacksystems.com'"
>       <somesh_gupta@silverbacksystems.com>, "BURBRIDGE,MATTHEW
>       (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>, "'Binford,
>       Charles'" <CBinford@pirus.com>, ips@ece.cmu.edu
> cc:
> Subject:  RE: Addition of CmdSN in Data-out PDU
>
>
>
> I have two small questions that would help me understand this.
>
> How do you know that unsolicited data is expected?  By nature,
> unsolicited data is received as a "maximum surprise" which can
> only be determined after unpacking and parsing at least a
> part of the command structure, possibly even including the
> SCSI command (although there are some hints contained in the
> command header information).
>
> How can you constrain a generic system to posting the
> unsolicited data in the same order that the commands were
> emitted?  In general, I would have expected the system to
> be emitting command followed immediately by the corresponding
> unsolicited data.  If that is not the case, it means that there
> was a delay in obtaining the unsolicited data for transfer and
> that the delay was sufficient to allow the insertion of commands.
> If the delay is that large (and probably variable), the enforcement
> of transfer of unsolicited data in the same order as the
> commands are emitted seems to me to be a significant challenge,
> and certainly shouldn't be required as normal behavior.  While it
> would make things simpler for targets (already challenged by
> unsolicited data), it seems to me that it would make things
> much more complex for initiators.
>
> Bob
>
> > -----Original Message-----
> > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com]
> > Sent: Friday, October 12, 2001 2:51 PM
> > To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Binford, Charles';
> > ips@ece.cmu.edu
> > Subject: RE: Addition of CmdSN in Data-out PDU
> >
> >
> > Matthew,
> >
> > Since the unsolicited data does not follow the
> > command, you need the link list. But since the
> > unsolicited data must be sent in the same order
> > as the commands, a link list is enough.
> >
> > Let us say that you have 8 commands. The ones
> > for which we expect unsolicted data are marked
> > as Cnn(ud). And I have marked the unsolicted
> > data PDUs as UD(nn). The (nn) with data implies
> > that it is implicit and not actually carried with
> > the PDU itself.
> >
> > C01(ud) C02(ud) C03(ud) C04 C05 C06(ud) UD(01) --->
> > --> C07(ud) UD(02) UD(03) D04 D04 D05 D05 UD(06) --->
> > --> UD(07) C08(ud) UD(08) --->
> >
> > After the target receives the command C01, C02, and C03
> > for which it expects unsolicited data, it puts them in
> > a link list. It also receives C04 and C05 for which
> > unsolicited data is not expected and they don't go
> > on the list. It then receives C06 for which unsolicited
> > data is expected, and it is added to then end of the list.
> > Then an unsolicited data PDU is received. It must go
> > with the command at the head of the list which is C01.
> > Use the ITT to make sure and you can then take C01 off
> > the list and so on.
> >
> > Somesh
> >
> >
> > > -----Original Message-----
> > > From: owner-ips@ece.cmu.edu
> > [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > > BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
> > > Sent: Friday, October 12, 2001 7:53 AM
> > > To: 'Binford, Charles'; ips@ece.cmu.edu
> > > Subject: RE: Addition of CmdSN in Data-out PDU
> > >
> > >
> > > Charles,
> > >
> > > As you have described the spec states that "that
> > unsolicited data MUST be
> > > sent in the same order as the commands".  This is not the same as
> > > unsolicited data must follow the command associated with
> > it:  For example:
> > >
> > > (Cx = SCSI Command PDU, Dx = The unsolicited data PDUs. The
> > x in all the
> > > example can be the ITT. It is not the CmdSN.
> > >
> > > This is allowed:
> > >
> > >   C1 C2 C3 D1 D2 C4 D3 D4
> > >
> > > and the target will have to use the ITT to associate the
> > data with the
> > > command.
> > >
> > > Matthew
> > >
> > >
> > > -----Original Message-----
> > > From: Binford, Charles [mailto:CBinford@pirus.com]
> > > Sent: Friday, October 12, 2001 2:50 PM
> > > To: ips@ece.cmu.edu
> > > Subject: RE: Addition of CmdSN in Data-out PDU
> > >
> > >
> > > I have not verify version 8 is still the same, but version 07-97
> > > had a rule
> > > that unsolicited data MUST be sent in the same order as the
> > commands.
> > >
> > > Thus, there is no need for a search on the ITT.  The target
> > just needs to
> > > keep of linked list of I/Os waiting on unsolicited data.
> > New commands are
> > > added to the tail, any unsolicited data *should* be associated
> > > with the I/O
> > > at the head of the list.  The ITT is used as a sanity check and
> > > you're done.
> > >
> > > What am I missing?
> > >
> > > Charles Binford
> > > Pirus Networks
> > > 316.315.0382 x222
>
>
>
>
>
>
>
>