SORT BY:

LIST ORDER
THREAD
AUTHOR
SUBJECT


SEARCH

IPS HOME


    [Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

    Re: iSCSI: Markers



    
    OK, Julian,
    I buy that argument.  I think that you have made the point that the value
    of Markers is Greater then I was giving credit for, in iSCSI/TOE integrated
    HBAs that are operating in CRC mode.   That is great, since I am afraid we
    will not see the Framing stuff for some time.
    
    So the questions are which form of markers is the best.
    
    My vote comes down on the side of Fixed Interval Markers (FIM). For the
    following reasons:
    
    On the SW sending side, FIM works with low overhead whether CRC is being
    used or not.  COWS, is a dramatic overhead unless the SW is performing CRC
    computation anyway.
    
    The COWS has a built in conflict with the Pipeline HW HBA, that has two
    different needs regarding the direction of the Pointers when talking to
    another partner  HW HBA that uses a Pipeline approach.
    
    .
    .
    .
    John L. Hufferd
    Senior Technical Staff Member (STSM)
    IBM/SSG San Jose Ca
    Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
    Home Office (408) 997-6136, Cell: (408) 499-9702
    Internet address: hufferd@us.ibm.com
    
    
    Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 01/07/2002 05:34:13 AM
    
    Sent by:    owner-ips@ece.cmu.edu
    
    
    To:    ips@ece.cmu.edu
    cc:
    Subject:    Re: iSCSI: Markers
    
    
    
    
    John,
    
    The markers are supposed to help you get fat to the next PDU and start
    assembling and placing it.
    It does not alleviate the need for a PDU assembly buffer - that you need
    anyhow - it alleviates the need to keep
    TCP packets along when you have lost an iSCSI PDU header.
    
    Once you have an iSCSI PDU header you need only the (as short as you want)
    PDU reassembly.
    
    If you do not have an iSCSI header you have potentially to keep around an
    RTT dependent amount of data.
    Markers are here to reduce the later to an amount that is not a function of
    Bandwidth*RTT.
    
    Regards,
    Julo
    
    
                                                                              
        John                      To:        Julian Satran/Haifa/IBM@IBMIL    
        Hufferd@IB                cc:        ips@ece.cmu.edu                  
        MUS                       From:        John Hufferd/San               
                          Jose/IBM@IBMUS                                      
        07-01-02                  Subject:        Re: iSCSI: MarkersLink      
        12:07                                                                 
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
    
    
    
    Julian,
    There are two different issues with regards to CRC.
    
    One is on the sender side, and if Markers are done in Software, of course
    it matters if CRC is done, since if the sender does not use CRC, the cost
    of COWS is very high.
    
    The second point is about the Receiving Side.  I did not understand your
    point, so I would appreciate you expanding on your comment that the Target
    Side would still get value out of Markers even if they use CRC.  I can
    understand that there would be some additional value, especially in TCP
    Segment errors.  By permitting some additional TCP Segments to arrive
    during the retry, but most of the value is lost because the Full Reassembly
    Buffer is needed for CRC.  So now that I confess, that I do not completely
    understand your point, perhaps you could step me through your view of the
    value in the presents of CRC on a HW HBA/Chip.   I think it is important
    that we are able to quantify the value.
    
    With regards to iWARP, I was using that name as an alias for the technique
    explained in the framing Draft (at
    http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-tcp-ulp-frame-01.txt).
    But except for the last part of COWS, there is little resemblance between
    that and COWS.  The framing does NOT use Word replacement.  They only have
    a 8 byte header in common.
    
    That brings me to another point.  If FIM needs to double its Markers (two
    words instead of one), would not COWS also need to double its Header
    Markers, for the same reason?
    
    
    .
    .
    .
    John L. Hufferd
    Senior Technical Staff Member (STSM)
    IBM/SSG San Jose Ca
    Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
    Home Office (408) 997-6136, Cell: (408) 499-9702
    Internet address: hufferd@us.ibm.com
    
    
    Sent by:        owner-ips@ece.cmu.edu
    
    To:        ips@ece.cmu.edu
    cc:
    Subject:        Re: iSCSI: Markers
    
    
    
    
    Some comments in text - Julo
    
    
                                                                              
         John Hufferd/San                                                     
         Jose/IBM@IBMUS                  To:        ips@ece.cmu.edu           
         Sent by:                        cc:                                  
         owner-ips@ece.cmu               Subject:        iSCSI: Markers       
         .edu                                                                 
                                                                              
         06-01-02 23:57                                                       
                                                                              
                                                                              
    
    
    
    When we were in Salt Lake City, we decided to hold a discussion on the
    Reflector regarding Markers.  I think it is probably time to hold that
    discussion.  Therefore, I will start it out.
    
    First I want to thank Julian for going to the trouble to document the
    proposal for COWS.  This now gives us two Marker proposals to consider.
    They are the Fixed Interval Markers (FIM) and the Constant Overhead Word
    Stuffing (COWS) proposals.
    
    Having said that, lets examine each one and determine their usefulness.
    
    1. Markers (FIM or COWS) are only needed by Hardware HBAs that are
    attempting to limit the amount of Reassembly Buffers they need on-board.
    This will always be a probabilistic determination by the vendors, but the
    purpose of Markers are to hold down the amount of total RAM needed,
    especially when factored into a probabilistic determination.
    2. The use of Markers are negotiated per connection, and per direction.  It
    is therefore possible for a software implementation to send markers, but
    not have to accept them.
    +++ that is specially important for software implementations ++
    3. When CRC Digest are negotiated to be used, it does no good to send out
    Markers since the hardware side will have to have Reassembly Buffers to
    compute the CRC Digest.  Therefore, Markers do not help anything when CRCs
    are used on a connection.
    +++ That is not correct - Framing mechanisms are used to avoid an RTT
    related size for the receive buffer not avoinding  a one-PDU buffer. Both
    FIM and COWS will help reducing HBA ememory requirements even when using
    CRC.  When Using CRC a PDU reassembly buffer is unavoidale but it can be
    limited by the MaxRcvPDU. +++
    
    4. FIM can be implemented easily in software, and with an out going
    Scatter/Gather technique could send PDUs across a network to a Hardware HBA
    destination without requiring additional moves or touching each byte.
    5. Software iSCSI implementations should always negotiate NOT to receive
    markers since Markers do not help the SW in any way.
    6. Hardware HBAs can also implement FIM easily, and HW can do it easily in
    both directions.  Therefore, if FIM is the approved Marker proposal, a HW
    HBA should send FIM whenever it sends iSCSI Commands and/or Data to other
    HW HBAs (assuming iWARP is not an available option), and the other HW HBAs
    wants them.
    7. I do not foresee many installations using CRC Digests with Laptops and
    Desktops, since the overhead will probably be noticeable, and most
    installations do not judge Desktop and Laptop data to be key corporate
    assets (I am not saying that opinion is right, or that it is Universal,
    just that is a prevalent opinion).
    +++CRCs are not relevant +++
    8. A single software node would not be fast enough to cause a significant
    load or a significant  memory requirement for the Reassembly Buffers.
    However, the combination of many (perhaps Desktops and Laptops spread
    across a real or virtual campus) can bring on a large load and the need for
    many Reassembly Buffers.  The resultant amount of small Reassembly Buffers
    could make the Total requirement very High (thus raising the cost of the
    HBA).
    9. Given the need for Desktops and Laptops to avoid noticeable degradation,
    and the probability that they will not be using CRC, means that the use of
    FIM is a compatible option when working across a network from a SW Node to
    an iSCSI HW HBA implemented Storage Controller.
    10. COWS was also designed to be easily implemented in Software and in
    Hardware.
    11. COWS will require the iSCSI Software implementations to touch each
    byte, as it looks for matches to its Framing Pattern within the PDUs.
    +++ except when implemented within the CRC, checksum where no ADDITIONAL
    touch is involved +++
    12. When CRC-32 digests are being computed, the Software will have to touch
    ever byte anyway, so the additional overhead to look for matches to the
    Framing Pattern is negligible.  On the other hand FIM also works well with
    SW CRC-32 Digest computation.
    13. But the premies I set above (which is clearly up to debate), is that
    Desktops and Laptops will not usually compute the CRC Digest.
    14. COWS has the ability to have its pointing Markers, which point to the
    Framing Pattern Matches, point either forward or backwards.  When being
    processed by software, the direction does not matter.  So since the Markers
    should only be used on outgoing PDUs, the approprate direction of the
    pointers is what ever the destination needs.
    15. Many vendors, especially at the higher speeds, are implementing a pipe
    line design.  The pipeline receive process, will most likely want the COWS
    pointers to be forward pointing (pointing in the direction of the end of
    the PDU).  Since the SW does not care, forward pointers should be fine.
    16. If the HW Pipelining iSCSI HBA is to send COWS Markers to another HW
    Pipelining HBA, there may be a problem.  The outgoing pipeline would prefer
    to have the pointers pointing backwards, but the receiving pipeline would
    prefer to have the pointers going forwards.  Therefore with COWS, either
    the sending or the receiving HW Pipelining HBA will be unhappy and pipeline
    design will get very much more complicated.
    17. Everyone likes iWARP better, but it requires changes to the Host SW
    TCP/IP stacks in the case of SW Initiators.  Since these SW installations
    are probably going to be Desktops and Laptops, these systems will very
    likely have some non current version of the OS, (such as windows 2k, ME,
    etc.).  Hence it is unlikely that they will have iWARP in the majority of
    the SW TCP/IP Stacks).
    +++iWARP needs framing as well and I would guess the technique will be
    close to what COWS is.  Future convergence speaks for COWS+++
    18. If, for some reason, a Server uses Software iSCSI, it will probably
    have the latest and greatest OS software, and if that is true, and if iWARP
    is approved by the IETF then it will probably have the TCP/IP extensions
    that are required by iWARP.  It will therefore use iWARP instead of any
    Marker solution.
    19. If the Server uses Software iSCSI,  and if iWARP is not approved by the
    IETF, then COWS or FIM are both  reasonable choices  for the Server, if the
    Server also uses the CRC option.  If CRC is not used on the Server, FIM is
    a better match with the Server SW iSCSI.
    19. iWARP is probably going to stay in IETF "experimental mode" for some
    time into the future.  And without iWARP being specified by iSCSI as -- at
    least, "MAY Implement" and "MAY use", the HW to HW mode will still need a
    Marker solution.
    20. Therefore, we are left with Markers as the only technique we can count
    on to aid and assist the iSCSI HW HBAs, keep their "on-board" RAM
    requirements to a minimum.
    
    Based on the above I come to the following conclusions:
    1. Markers are needed.
    2. FIM works best with the probable SW environment, and has the lowest
    overhead (since they will probably not use CRC Digests).
    +++CRCs are not relevant+++
    3. Since if CRC is used, the HW needs its Reassembly Buffers anyway, so
    there is no reason to use COWS, or any type of Marker.
    +++ again the same confusion betwen the PDU buffer and the RTT related
    holding are in the absence of Markers+++
    4. FIM can work well between HW HBAs without significant impact to pipeline
    design.
    +++ Somebody already said that FIMs are like democracy - imperfect, ugly
    but work+++
    +++ However COWS are not that much heavier if carefully implemented and
    might get us faster to converge on a framing model+++
    
    Therefore, I recommend that the Draft continue to define FIM.  Also, I
    recommend that the FIM type of Markers be "MUST Implement" on the Out-Going
    direction, and "MAY Implement" on the incoming direction".   Further, I
    recommend the Draft say that FIM "MAY be used" in any direction, depending
    on the negotiations.
    
    .
    .
    .
    John L. Hufferd
    Senior Technical Staff Member (STSM)
    IBM/SSG San Jose Ca
    Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
    Home Office (408) 997-6136, Cell: (408) 499-9702
    Internet address: hufferd@us.ibm.com
    
    
    
    
    
    
    
    
    
    
    
    


Home

Last updated: Tue Jan 08 13:17:51 2002
8309 messages in chronological order