SORT BY:

LIST ORDER
THREAD
AUTHOR
SUBJECT


SEARCH

IPS HOME


    [Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

    RE: iSCSI: Out of order commands



    John,
    
    > -----Original Message-----
    > From: John Hufferd [mailto:hufferd@us.ibm.com]
    > Sent: Thursday, November 08, 2001 5:19 PM
    > To: somesh_gupta@silverbacksystems.com
    > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
    > Subject: RE: iSCSI: Out of order commands
    >
    >
    >
    > I understand your point, however, their are several approaches that I and
    > others have used to mitigate this issue.  We face that problem with Multi
    > Processing all the time.
    >
    > One technique that I have seen use that is similar to what you describe is
    > to allocate a set of buffers to each processor (in this case an HBA), and
    > when they are used up, and only then, the go back to the Buffer Mother, to
    > allocate another Buffer Set.  In this way the Processors (HBA) run mostly
    > independent, but once in awhile get a new supply.  There are other
    > techniques also, but that one clearly works wells and is very simple.
    
      I agree with the mechanism above. So here is where the problem is.
      Let us take a simplification of M adapters and 1 connection per
      adapter. Let us also assume that the N buffers are given to each
      adapter.
    
      Total number of buffers used - M * N
      Buffers available to each adapter - N
    
      How large of a command window should be advertised in the case of
      a session wide window.
    
      If the window is N, then we are under-utilizing the buffers but we
      will never drop commands.
    
      If we advertise M * N, and the initiator decides to send commands
      in an unbalanced manner (because 1 connection is working better),
      then we have a potential problem where the one adapter uses up
      its allocation - In the meantime the system will provide it some
      more (but not enough),
      and there is an imbalance leading to a situation where commands
      may have to be dropped. (this could also happen if the system
      did not allocate buffers evenly e.g. in a scenario where the
      adapters are not using buffers at the same rate).
    
      Now we can try to find a balance in the middle, but again there
      is a loss of efficiency on one axis or the other.
    
    >
    > Therefore, I do not see the Session Window as a problem in the protocol.
    >
    > .
    > .
    > .
    > John L. Hufferd
    > Senior Technical Staff Member (STSM)
    > IBM/SSG San Jose Ca
    > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
    > Home Office (408) 997-6136, Cell: (408) 499-9702
    > Internet address: hufferd@us.ibm.com
    >
    >
    > "Somesh Gupta" <somesh_gupta@silverbacksystems.com> on 11/08/2001 04:57:40
    > PM
    >
    > Please respond to <somesh_gupta@silverbacksystems.com>
    >
    > To:   John Hufferd/San Jose/IBM@IBMUS
    > cc:   "Robert D. Russell" <rdr@mars.iol.unh.edu>, Julian
    >       Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
    > Subject:  RE: iSCSI: Out of order commands
    >
    >
    >
    > I think we may be talking about different things here.
    > Yes, there are memories that are "on the iSCSI HBA"
    > (in the case where everything is on a single board,
    > this would be the memories that are controlled by
    > only the iSCSI ASIC or whatever) - and these memories
    > are not shared among multiple HBAs. These may be used
    > for any book-keeping purposes, contexts, eddy buffering
    > etc etc by the adapter.
    >
    > Then there is memory pool that is owned by the "system".
    > Adapters will DMA commands/data/responses directly into
    > and out of this memory pool which is managed by the "system".
    > However to prevent adapters from running over each other,
    > and over the "system" processor, producer/consumer
    > mechanisms have to be implemented.
    >
    > This is necessary to maintain order on the system. So
    > on a long-term basis, the memory in the system is shared
    > across the HBAs - absolutely yes. But at any given
    > instance in time, it is partitioned among various
    > usages.
    >
    > So all in all I agree with you. The only point of difference
    > is that it is not possible that a given buffer (part of
    > a buffer pool) is accessible to multiple adapters at
    > the same instance in time. It is a necessary condition to
    > prevent corruption and chaos.
    >
    > > -----Original Message-----
    > > From: John Hufferd [mailto:hufferd@us.ibm.com]
    > > Sent: Thursday, November 08, 2001 4:40 PM
    > > To: somesh_gupta@silverbacksystems.com
    > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
    > > Subject: RE: iSCSI: Out of order commands
    > >
    > >
    > >
    > > I guess I disagree with this design approach.  The buffers on the iSCSI
    > > HBAs are for Eddy buffers.  Once you have the iSCSI Header
    > should be able
    > > to place the data directly into a Shared buffer.  If you have a session
    > > that spans HBAs, that should work just fine.  But the buffers match the
    > > Session Window, and the Buffers need to be shared across the
    > HBAs, not in
    > > the HBAs.  Again, the HBAs should just have the Eddy Buffers.
    > >
    > > .
    > > .
    > > .
    > > John L. Hufferd
    > > Senior Technical Staff Member (STSM)
    > > IBM/SSG San Jose Ca
    > > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
    > > Home Office (408) 997-6136, Cell: (408) 499-9702
    > > Internet address: hufferd@us.ibm.com
    > >
    > >
    > > "Somesh Gupta" <somesh_gupta@silverbacksystems.com> on 11/08/2001
    > 04:31:29
    > > PM
    > >
    > > Please respond to <somesh_gupta@silverbacksystems.com>
    > >
    > > To:   John Hufferd/San Jose/IBM@IBMUS
    > > cc:   "Robert D. Russell" <rdr@mars.iol.unh.edu>, Julian
    > >       Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
    > > Subject:  RE: iSCSI: Out of order commands
    > >
    > >
    > >
    > > John,
    > >
    > > You are correct if all the connections are on
    > > a single adapter. In this case the buffer pool
    > > can be shared among all the connections on the
    > > adapter.
    > >
    > > However this is not true if the connections are
    > > on different adapters (the point being discussed).
    > > Think of there being a big buffer pool in the board
    > > into which multiple adapters/chips are "plugged" in.
    > > However, at any given point in time, these buffers
    > > are allocated (handed over through command queues or
    > > whatever) to each of the individual adapters. And this
    > > is where the problem arises.
    > >
    > > The assumption that adapters can share a typical host
    > > command buffer pool on their own without host intervention
    > > is incorrect.
    > >
    > > Somesh
    > >
    > > > -----Original Message-----
    > > > From: John Hufferd [mailto:hufferd@us.ibm.com]
    > > > Sent: Thursday, November 08, 2001 4:20 PM
    > > > To: somesh_gupta@silverbacksystems.com
    > > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
    > > > Subject: RE: iSCSI: Out of order commands
    > > >
    > > >
    > > >
    > > > Most folks I know have a buffer that applies to all the
    > > > connections, not to
    > > > just one buffer per connection.  I fail to see the problem for most
    > > > implimentations.  The window should apply to the session.
    > > >
    > > > .
    > > > .
    > > > .
    > > > John L. Hufferd
    > > > Senior Technical Staff Member (STSM)
    > > > IBM/SSG San Jose Ca
    > > > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
    > > > Home Office (408) 997-6136, Cell: (408) 499-9702
    > > > Internet address: hufferd@us.ibm.com
    > > >
    > > >
    > > > "Somesh Gupta" <somesh_gupta@silverbacksystems.com>@ece.cmu.edu on
    > > > 11/08/2001 02:22:30 PM
    > > >
    > > > Please respond to <somesh_gupta@silverbacksystems.com>
    > > >
    > > > Sent by:  owner-ips@ece.cmu.edu
    > > >
    > > >
    > > > To:   "Robert D. Russell" <rdr@mars.iol.unh.edu>
    > > > cc:   Julian Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
    > > > Subject:  RE: iSCSI: Out of order commands
    > > >
    > > >
    > > >
    > > > Comments below
    > > >
    > > > > -----Original Message-----
    > > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu]
    > > > > Sent: Thursday, November 08, 2001 11:46 AM
    > > > > To: Somesh Gupta
    > > > > Cc: Julian Satran; ips@ece.cmu.edu
    > > > > Subject: RE: iSCSI: Out of order commands
    > > > >
    > > > >
    > > > >
    > > > > Somesh:
    > > > >
    > > > > > > It seems to me that if a target can only handle x new commands,
    > > > > > > then its queueing capacity is x, and in the SCSI Response PDU it
    > > > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the
    > > > > > > formula in section 2.2.2.1.  This in turn controls the number of
    > > > > > > commands the initiator can send, and thus prevents the incoming
    > > > > > > commands from overfilling the target's queue.
    > > > > >
    > > > > >   This is actually a weakness of the multiple connections over
    > > > > >   multiple adapters model (it is not related to ordering).
    > > > > >   The target advertises a command window on a session wide basis.
    > > > > >   At the same time, it has to provide the resource to the adapter
    > > > > >   to be able to DMA those commands to. Since there is no
    > restriction
    > > > > >   on which connection the commands may be received, either the
    > > > > >   target has to allocate the max resources needed to each adapter
    > > > > >   (thus committing n times the resources actually required), or
    > > > > >   has to "lie" (it would not be a complete lie) which could
    > > > > >   result in running out of resources. One way to fix that would be
    > > > > >   to have the credit on a per connection basis.
    > > > >
    > > > > If I understand you correctly, you are saying that deadlock can
    > > > > occur even if we enforce ordered delivery by initiators and even
    > > > > if the advertisements are correct and even if both parties obey
    > > > > the advertisements.
    > > >
    > > >   I would let Julian enlighten us on that, or whether it just
    > > >   leads to wholesale command drops and retries, or TASK SET FULL
    > > >   is returned (is TASK SET FULL a valid response in this case
    > > >   for a LINKED task when it is not the first command?)
    > > >
    > > > >
    > > > > In other words, doesn't this mean that the standard can lead to a
    > > > > trap in implementing targets and that, in fact, the advertisements
    > > > > really should be on a per connection basis?
    > > > > However, what would be the consequences for initiators if
    > > > > the advertisements were on a per-connection basis?
    > > >
    > > >   There is really no issue with adveritisement on a per
    > > >   connection basis. It leads to a protocol change. However, it
    > > >   also allows a much more performant flow of commands.
    > > >
    > > > >
    > > > >
    > > > > Thanks,
    > > > >
    > > > > Bob Russell
    > > > > InterOperability Lab
    > > > > University of New Hampshire
    > > > > rdr@iol.unh.edu
    > > > > 603-862-3774
    > > > >
    > > > >
    > > > >
    > > >
    > > >
    > > >
    > > >
    > >
    > >
    > >
    > >
    >
    >
    >
    >
    >
    
    


Home

Last updated: Thu Nov 08 21:17:38 2001
7677 messages in chronological order