RE: iSCSI: Out of order commands

["Somesh Gupta" <somesh_gupta@silverbacksystems.com>]

I think we may be talking about different things here.
Yes, there are memories that are "on the iSCSI HBA"
(in the case where everything is on a single board,
this would be the memories that are controlled by
only the iSCSI ASIC or whatever) - and these memories
are not shared among multiple HBAs. These may be used
for any book-keeping purposes, contexts, eddy buffering
etc etc by the adapter.

Then there is memory pool that is owned by the "system".
Adapters will DMA commands/data/responses directly into
and out of this memory pool which is managed by the "system".
However to prevent adapters from running over each other,
and over the "system" processor, producer/consumer
mechanisms have to be implemented.

This is necessary to maintain order on the system. So
on a long-term basis, the memory in the system is shared
across the HBAs - absolutely yes. But at any given
instance in time, it is partitioned among various
usages.

So all in all I agree with you. The only point of difference
is that it is not possible that a given buffer (part of
a buffer pool) is accessible to multiple adapters at
the same instance in time. It is a necessary condition to
prevent corruption and chaos.

> -----Original Message-----
> From: John Hufferd [mailto:hufferd@us.ibm.com]
> Sent: Thursday, November 08, 2001 4:40 PM
> To: somesh_gupta@silverbacksystems.com
> Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
> Subject: RE: iSCSI: Out of order commands
>
>
>
> I guess I disagree with this design approach.  The buffers on the iSCSI
> HBAs are for Eddy buffers.  Once you have the iSCSI Header should be able
> to place the data directly into a Shared buffer.  If you have a session
> that spans HBAs, that should work just fine.  But the buffers match the
> Session Window, and the Buffers need to be shared across the HBAs, not in
> the HBAs.  Again, the HBAs should just have the Eddy Buffers.
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: hufferd@us.ibm.com
>
>
> "Somesh Gupta" <somesh_gupta@silverbacksystems.com> on 11/08/2001 04:31:29
> PM
>
> Please respond to <somesh_gupta@silverbacksystems.com>
>
> To:   John Hufferd/San Jose/IBM@IBMUS
> cc:   "Robert D. Russell" <rdr@mars.iol.unh.edu>, Julian
>       Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
> Subject:  RE: iSCSI: Out of order commands
>
>
>
> John,
>
> You are correct if all the connections are on
> a single adapter. In this case the buffer pool
> can be shared among all the connections on the
> adapter.
>
> However this is not true if the connections are
> on different adapters (the point being discussed).
> Think of there being a big buffer pool in the board
> into which multiple adapters/chips are "plugged" in.
> However, at any given point in time, these buffers
> are allocated (handed over through command queues or
> whatever) to each of the individual adapters. And this
> is where the problem arises.
>
> The assumption that adapters can share a typical host
> command buffer pool on their own without host intervention
> is incorrect.
>
> Somesh
>
> > -----Original Message-----
> > From: John Hufferd [mailto:hufferd@us.ibm.com]
> > Sent: Thursday, November 08, 2001 4:20 PM
> > To: somesh_gupta@silverbacksystems.com
> > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
> > Subject: RE: iSCSI: Out of order commands
> >
> >
> >
> > Most folks I know have a buffer that applies to all the
> > connections, not to
> > just one buffer per connection.  I fail to see the problem for most
> > implimentations.  The window should apply to the session.
> >
> > .
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/SSG San Jose Ca
> > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > Home Office (408) 997-6136, Cell: (408) 499-9702
> > Internet address: hufferd@us.ibm.com
> >
> >
> > "Somesh Gupta" <somesh_gupta@silverbacksystems.com>@ece.cmu.edu on
> > 11/08/2001 02:22:30 PM
> >
> > Please respond to <somesh_gupta@silverbacksystems.com>
> >
> > Sent by:  owner-ips@ece.cmu.edu
> >
> >
> > To:   "Robert D. Russell" <rdr@mars.iol.unh.edu>
> > cc:   Julian Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
> > Subject:  RE: iSCSI: Out of order commands
> >
> >
> >
> > Comments below
> >
> > > -----Original Message-----
> > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu]
> > > Sent: Thursday, November 08, 2001 11:46 AM
> > > To: Somesh Gupta
> > > Cc: Julian Satran; ips@ece.cmu.edu
> > > Subject: RE: iSCSI: Out of order commands
> > >
> > >
> > >
> > > Somesh:
> > >
> > > > > It seems to me that if a target can only handle x new commands,
> > > > > then its queueing capacity is x, and in the SCSI Response PDU it
> > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the
> > > > > formula in section 2.2.2.1.  This in turn controls the number of
> > > > > commands the initiator can send, and thus prevents the incoming
> > > > > commands from overfilling the target's queue.
> > > >
> > > >   This is actually a weakness of the multiple connections over
> > > >   multiple adapters model (it is not related to ordering).
> > > >   The target advertises a command window on a session wide basis.
> > > >   At the same time, it has to provide the resource to the adapter
> > > >   to be able to DMA those commands to. Since there is no restriction
> > > >   on which connection the commands may be received, either the
> > > >   target has to allocate the max resources needed to each adapter
> > > >   (thus committing n times the resources actually required), or
> > > >   has to "lie" (it would not be a complete lie) which could
> > > >   result in running out of resources. One way to fix that would be
> > > >   to have the credit on a per connection basis.
> > >
> > > If I understand you correctly, you are saying that deadlock can
> > > occur even if we enforce ordered delivery by initiators and even
> > > if the advertisements are correct and even if both parties obey
> > > the advertisements.
> >
> >   I would let Julian enlighten us on that, or whether it just
> >   leads to wholesale command drops and retries, or TASK SET FULL
> >   is returned (is TASK SET FULL a valid response in this case
> >   for a LINKED task when it is not the first command?)
> >
> > >
> > > In other words, doesn't this mean that the standard can lead to a
> > > trap in implementing targets and that, in fact, the advertisements
> > > really should be on a per connection basis?
> > > However, what would be the consequences for initiators if
> > > the advertisements were on a per-connection basis?
> >
> >   There is really no issue with adveritisement on a per
> >   connection basis. It leads to a protocol change. However, it
> >   also allows a much more performant flow of commands.
> >
> > >
> > >
> > > Thanks,
> > >
> > > Bob Russell
> > > InterOperability Lab
> > > University of New Hampshire
> > > rdr@iol.unh.edu
> > > 603-862-3774
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>