[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

    Re: iSCSI/iWARP drafts and flow control

    • To: Caitlin Bestler <>
    • Subject: Re: iSCSI/iWARP drafts and flow control
    • From: Mike Ko <>
    • Date: Sun, 27 Jul 2003 12:21:31 -0700
    • Cc:
    • Content-Type: text/plain; charset="us-ascii"
    • Delivered-To:
    • Delivered-To:
    • Delivered-To:
    • Delivered-To:
    • Importance: Normal
    • Sender:

    From sec. of the iSCSI spec, "Asychronoous Messages are used to 
    carry SCSI asyncrhonous events (AEN) and iSCSI asynchronous messages." 
    Sec.10.9.1 lists the codes for Asyncrhonous Messages which include 0) SCSI 
    Asynchronous Event, 1) target requests Logout, 2) target indicates it will 
    drop the connection, 3) target indicates it will drop all connections of 
    this session, and 4) target requests parameter negotiation.  Since the 
    iSER layer can estimate the number of Asynchronous Messages needed in a 
    typical situation for the AsyncEvent codes listed in the iSCSI spec, it 
    can determine how much to overprovision when a dedicated receive queue is 
    used.  Additionally, if one is concerned about the atypical case, the 
    optional features of graceful handling and the shared receive queue as you 
    have pointed out can be used.  So it doesn't seem to me that flow control 
    for Asynchronous Messages are needed.  Can you provide an example where an 
    unbounded number of Asynchronous Messages would need to be sent?
    Mike Ko
    IBM Almaden Research
    San Jose, CA 95120
    To:     Mike Ko <>
    Subject:        Re: iSCSI/iWARP drafts and flow control
    On Saturday, July 26, 2003, at 03:57 PM, Mike Ko wrote:
    > In iSER, we expect the flow control to be regulated by the Command
    > Numbering mechanism in iSCSI.  In other words, since the queuing
    > capacity
    > of the receiving iSCSI layer is MaxCmdSN - ExpCmdSN + 1, the receiving
    > iSER layer can use this information to determine the minimum number of
    > untagged buffers.  In addition, it needs to provision a sufficient
    > number
    > of untagged buffers to allow enough time for the iSER layer to respond
    > to
    > incoming immediate commands, asynchronous messages, etc., and replenish
    > the buffers.  The use of a buffer pool shared across multiple
    > connections
    > will allow the iSER layer to replenish the buffers on a statistical
    > basis.
    iSCSI does indeed provide control for untagged messages that
    have CmdSNs. The issue relates to Asynchronous Messages.
    Is there any protocol restraint on the number of Asynchronous
    Messages that can be sent? The only one that I could find is in
    section where it is limited to an "implementation defined
    constant that MUST NOT exceed 2**31-1".
    You cannot know an implementation defined constant of your peer,
    unless the protocol explicitly communicates it. So the only ULP
    constraint on the number of in-flight Asynchronous messages is
    2**31-1. That constraint already existed in iWARP (the Message
    Sequence Number has the same need to determine which of two 32
    bit numbers that wrap-around is "later").
    This is an important change from iSCSI over a conventional
    transport protocol. The asynchronous messages would be flow
    controlled by the transport itself. The Data Sink is expected
    to advertise the amount of in-flight data that it is willing
    to accept.
    It is important to separate buffering from flow control. Flow
    control deals with what is legal to send. Buffering is strictly
    a local issue. Even with conventional transports, there is no
    requirement that every buffer byte advertised on a TCP connection
    or SCTP association be pre-allocated and reserved in advance.
    How a host manages its receive buffers is a local issue. The
    protocol issue is what it tells its peer that the peer is
    authorized to send.
    Aside from the theoretical issues (such as definitions of "reliable
    transport" and "flow control") there is a very nitty gritty issue
    that buffer pools are an *optional* feature even under the RDMAC
    proposed verbs. You also cannot "share" resources across connections
    if there is only a single connection to deal with.
    To achieve the goal of allowing iSER to work on a generic RDMA
    card it must work with a *simple* Receive Queue.  That requires
    that the Data Sink be able to predict the maximum number of
    untagged messages that can be in-flight from a peer that is
    following the protocol.
    This actually would not be that hard to deal with. Several of
    the Asynchronous Messages are inherently unique. There is no
    reason to send two messages announcing the intent to terminate
    a connection over a reliable connection.
    If you simply add a constraint on the number of Asynchronous
    Messages that may be sent without confirmation that they have
    been processed there is no problem. iSER becomes a reliable
    protocol. Peers are not required to estimate how fast the
    other side can process messages. (And, to hit the theoretical
    point, if you have to *estimate* how fast your peer operates
    then you are no longer using a *reliable* protocol).
    There are already "credits" for CmdSN-bearing untagged message.
    The methods for establishing and modifying these credits already
    exist in iSCSI so there is no need for iSER to do anything
    other than be aware of the value.
    iSER then has to factor in additional credits for Asychs.
    This could be a ULP specified constant, or it could be
    negotiated just as is proposed for RDMA Read credits.
    Credits would be drained by sending. All that is required
    is the addition of of a rule on how credits are restored.
    I would suggest using reply to a *later* CmdSN and/or
    a command that explicitly waits for all prior Asynch
    Messages to have been processed.


Last updated: Tue Aug 05 12:46:09 2003
12771 messages in chronological order