RE: iSCSI: Addition of CmdSN in Data-out PDU

["Rod Harrison" <rod.harrison@windriver.com>]

Julian,

	I think you might have misunderstood what I meant. I was not
advocating we change the spec in favour of relaxing the data ordering
requirements. I was just point out there is a problem for the
initiator to solve in order maintain correct ordering, and that I
believe it is not a terribly difficult one.

	- Rod

-----Original Message-----
From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
Julian Satran
Sent: Sunday, October 14, 2001 9:30 PM
To: ips@ece.cmu.edu
Subject: RE: Addition of CmdSN in Data-out PDU



Immediate data are not mandatory (even if enabled). Having one or ten
DMAs
does not change the fact that they share a TCP "pipe" and a decent
transmitter will ship the TCP data in order - otherwise it just throws
his
problems in the receiver's lap. Our additional requirement on order
should
make no difference.  And it would be time to change the name of the
thread.

Julo





                    "Rod Harrison"
                    <rod.harrison@wind       To:     John Hufferd/San
Jose/IBM@IBMUS, "Robert
                    river.com>                Snively"
<rsnively@Brocade.COM>
                    Sent by:                 cc:
<somesh_gupta@silverbacksystems.com>,
                    owner-ips@ece.cmu.        "BURBRIDGE,MATTHEW
(HP-UnitedKingdom,ex2)"
                    edu
<matthew_burbridge@hp.com>, "'Binford, Charles'"
                                              <CBinford@pirus.com>,
<ips@ece.cmu.edu>
                                             Subject:     RE: Addition
of CmdSN in Data-out PDU
                    13-10-01 13:12
                    Please respond to
                    "Rod Harrison"






           Even more common would be the DMA latency. As an initiator
I
would
send the command PDU as soon as I could if there were no immediate
data, then queue the DMA for the unsolicited data. Since there could
be many other DMAs queued ahead of the unsolicited data it is quite
possible that other commands, both read and write, will be sent before
the DMA completes.

           You are spot one when you say the delay in obtaining the
unsolicited
data may be large and variable. If there are multiple DMA engines it
does become a challenge for the initiator to ensure the correct
ordering of the unsolicited data. Note that since there may be
multiple unsolicited data PDUs in the same burst, and those may be
filled with separate DMAs the ordering problem propagates down to
within individual commands.

           I don't believe it is a particularly difficult problem for
the
initiator to solve, but I do agree things would be easier for the
initiator if the ordering requirement were removed. And more so if the
requirement to send in dataSN order were also removed, but this is
probably too much pain for the target.

           - Rod

-----Original Message-----
From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
John Hufferd
Sent: Saturday, October 13, 2001 1:01 AM
To: Robert Snively
Cc: 'somesh_gupta@silverbacksystems.com'; BURBRIDGE,MATTHEW
(HP-UnitedKingdom,ex2); 'Binford, Charles'; ips@ece.cmu.edu
Subject: RE: Addition of CmdSN in Data-out PDU



Robert,
I think that an Initiator being able to send a waiting Read command,
without having to wait for many large write segments -- that are being
sent
(as unsolicited data) -- is very useful.  And that would mean, the
unsolicited data is waiting to be sent until the Read Commands are
sent.
This might be a very frequent case.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: hufferd@us.ibm.com


Robert Snively <rsnively@Brocade.COM>@ece.cmu.edu on 10/12/2001
03:56:06 PM

Sent by:  owner-ips@ece.cmu.edu


To:   "'somesh_gupta@silverbacksystems.com'"
      <somesh_gupta@silverbacksystems.com>, "BURBRIDGE,MATTHEW
      (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>, "'Binford,
      Charles'" <CBinford@pirus.com>, ips@ece.cmu.edu
cc:
Subject:  RE: Addition of CmdSN in Data-out PDU



I have two small questions that would help me understand this.

How do you know that unsolicited data is expected?  By nature,
unsolicited data is received as a "maximum surprise" which can
only be determined after unpacking and parsing at least a
part of the command structure, possibly even including the
SCSI command (although there are some hints contained in the
command header information).

How can you constrain a generic system to posting the
unsolicited data in the same order that the commands were
emitted?  In general, I would have expected the system to
be emitting command followed immediately by the corresponding
unsolicited data.  If that is not the case, it means that there
was a delay in obtaining the unsolicited data for transfer and
that the delay was sufficient to allow the insertion of commands.
If the delay is that large (and probably variable), the enforcement
of transfer of unsolicited data in the same order as the
commands are emitted seems to me to be a significant challenge,
and certainly shouldn't be required as normal behavior.  While it
would make things simpler for targets (already challenged by
unsolicited data), it seems to me that it would make things
much more complex for initiators.

Bob

> -----Original Message-----
> From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com]
> Sent: Friday, October 12, 2001 2:51 PM
> To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Binford, Charles';
> ips@ece.cmu.edu
> Subject: RE: Addition of CmdSN in Data-out PDU
>
>
> Matthew,
>
> Since the unsolicited data does not follow the
> command, you need the link list. But since the
> unsolicited data must be sent in the same order
> as the commands, a link list is enough.
>
> Let us say that you have 8 commands. The ones
> for which we expect unsolicted data are marked
> as Cnn(ud). And I have marked the unsolicted
> data PDUs as UD(nn). The (nn) with data implies
> that it is implicit and not actually carried with
> the PDU itself.
>
> C01(ud) C02(ud) C03(ud) C04 C05 C06(ud) UD(01) --->
> --> C07(ud) UD(02) UD(03) D04 D04 D05 D05 UD(06) --->
> --> UD(07) C08(ud) UD(08) --->
>
> After the target receives the command C01, C02, and C03
> for which it expects unsolicited data, it puts them in
> a link list. It also receives C04 and C05 for which
> unsolicited data is not expected and they don't go
> on the list. It then receives C06 for which unsolicited
> data is expected, and it is added to then end of the list.
> Then an unsolicited data PDU is received. It must go
> with the command at the head of the list which is C01.
> Use the ITT to make sure and you can then take C01 off
> the list and so on.
>
> Somesh
>
>
> > -----Original Message-----
> > From: owner-ips@ece.cmu.edu
> [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
> > Sent: Friday, October 12, 2001 7:53 AM
> > To: 'Binford, Charles'; ips@ece.cmu.edu
> > Subject: RE: Addition of CmdSN in Data-out PDU
> >
> >
> > Charles,
> >
> > As you have described the spec states that "that
> unsolicited data MUST be
> > sent in the same order as the commands".  This is not the same as
> > unsolicited data must follow the command associated with
> it:  For example:
> >
> > (Cx = SCSI Command PDU, Dx = The unsolicited data PDUs. The
> x in all the
> > example can be the ITT. It is not the CmdSN.
> >
> > This is allowed:
> >
> >   C1 C2 C3 D1 D2 C4 D3 D4
> >
> > and the target will have to use the ITT to associate the
> data with the
> > command.
> >
> > Matthew
> >
> >
> > -----Original Message-----
> > From: Binford, Charles [mailto:CBinford@pirus.com]
> > Sent: Friday, October 12, 2001 2:50 PM
> > To: ips@ece.cmu.edu
> > Subject: RE: Addition of CmdSN in Data-out PDU
> >
> >
> > I have not verify version 8 is still the same, but version 07-97
> > had a rule
> > that unsolicited data MUST be sent in the same order as the
> commands.
> >
> > Thus, there is no need for a search on the ITT.  The target
> just needs to
> > keep of linked list of I/Os waiting on unsolicited data.
> New commands are
> > added to the tail, any unsolicited data *should* be associated
> > with the I/O
> > at the head of the list.  The ITT is used as a sanity check and
> > you're done.
> >
> > What am I missing?
> >
> > Charles Binford
> > Pirus Networks
> > 316.315.0382 x222