[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

    Re: iSCSI/iWARP drafts and flow control

    On Saturday, July 26, 2003, at 03:57 PM, Mike Ko wrote:
    > In iSER, we expect the flow control to be regulated by the Command
    > Numbering mechanism in iSCSI.  In other words, since the queuing 
    > capacity
    > of the receiving iSCSI layer is MaxCmdSN - ExpCmdSN + 1, the receiving
    > iSER layer can use this information to determine the minimum number of
    > untagged buffers.  In addition, it needs to provision a sufficient 
    > number
    > of untagged buffers to allow enough time for the iSER layer to respond 
    > to
    > incoming immediate commands, asynchronous messages, etc., and replenish
    > the buffers.  The use of a buffer pool shared across multiple 
    > connections
    > will allow the iSER layer to replenish the buffers on a statistical 
    > basis.
    iSCSI does indeed provide control for untagged messages that
    have CmdSNs. The issue relates to Asynchronous Messages.
    Is there any protocol restraint on the number of Asynchronous
    Messages that can be sent? The only one that I could find is in
    section where it is limited to an "implementation defined
    constant that MUST NOT exceed 2**31-1".
    You cannot know an implementation defined constant of your peer,
    unless the protocol explicitly communicates it. So the only ULP
    constraint on the number of in-flight Asynchronous messages is
    2**31-1. That constraint already existed in iWARP (the Message
    Sequence Number has the same need to determine which of two 32
    bit numbers that wrap-around is "later").
    This is an important change from iSCSI over a conventional
    transport protocol. The asynchronous messages would be flow
    controlled by the transport itself. The Data Sink is expected
    to advertise the amount of in-flight data that it is willing
    to accept.
    It is important to separate buffering from flow control. Flow
    control deals with what is legal to send. Buffering is strictly
    a local issue. Even with conventional transports, there is no
    requirement that every buffer byte advertised on a TCP connection
    or SCTP association be pre-allocated and reserved in advance.
    How a host manages its receive buffers is a local issue. The
    protocol issue is what it tells its peer that the peer is
    authorized to send.
    Aside from the theoretical issues (such as definitions of "reliable
    transport" and "flow control") there is a very nitty gritty issue
    that buffer pools are an *optional* feature even under the RDMAC
    proposed verbs. You also cannot "share" resources across connections
    if there is only a single connection to deal with.
    To achieve the goal of allowing iSER to work on a generic RDMA
    card it must work with a *simple* Receive Queue.  That requires
    that the Data Sink be able to predict the maximum number of
    untagged messages that can be in-flight from a peer that is
    following the protocol.
    This actually would not be that hard to deal with. Several of
    the Asynchronous Messages are inherently unique. There is no
    reason to send two messages announcing the intent to terminate
    a connection over a reliable connection.
    If you simply add a constraint on the number of Asynchronous
    Messages that may be sent without confirmation that they have
    been processed there is no problem. iSER becomes a reliable
    protocol. Peers are not required to estimate how fast the
    other side can process messages. (And, to hit the theoretical
    point, if you have to *estimate* how fast your peer operates
    then you are no longer using a *reliable* protocol).
    There are already "credits" for CmdSN-bearing untagged message.
    The methods for establishing and modifying these credits already
    exist in iSCSI so there is no need for iSER to do anything
    other than be aware of the value.
    iSER then has to factor in additional credits for Asychs.
    This could be a ULP specified constant, or it could be
    negotiated just as is proposed for RDMA Read credits.
    Credits would be drained by sending. All that is required
    is the addition of of a rule on how credits are restored.
    I would suggest using reply to a *later* CmdSN and/or
    a command that explicitly waits for all prior Asynch
    Messages to have been processed.


Last updated: Tue Aug 05 12:46:09 2003
12771 messages in chronological order