RE: iSCSI: Recognizing Recovery R2Ts

[Nick Bellinger <nickb@attheoffice.org>]

On Mon, 2002-09-16 at 04:52, Julian Satran wrote:
> Nick,
> 
> In case you have ErrorRecoveryLevel >=1 and DataSequenceInOrder=Yes then 
> then either:
> 
> the target has only one outstanding R2T (to avoid at all price an 
> out-of-order R2T)
> or the initiator can go by an out-of-order R2T if it is a data-offset it 
> has "passed-over" (even if not touched). Please observe that an initiator 
> is aware of the gap in the R2T sequence when an R2T is lost if the target 
> has more than one R2T missing and the missing one is not the last. If the 
> missing one is the last (or only) there will be no violation of ordering.
> 
> Regards,
> Julo.
> The ordering rules require the offsets to be in non-decreasing order.
> 
>

Ok,  just to few more points to verify with regard to a lost R2T due to
a Header Digest error:

In the case of ErrorRecoveryLevel=>1, DataSequenceinOrder=No and
MaxOutStandingR2T=>1:

	Initiator is able to ascertain a lost R2T by receiving an
	R2TSN greater than expected,  and sends a SNACK of type R2T
	to request retransmission of the lost R2T. Target retransmits 	R2T with
identical values as the lost R2T.

In the case of ErrorRecoveryLevel=>1, DataSequenceinOrder=Yes and
MaxOutStandingR2T=1:

	Target will timeout waiting for Data OUT from the lost R2T
	and sends a Recovery R2T with identical values of the lost R2T.


Thanks again for your time gentlemen,
	
		Nicholas Bellinger - Pyxtechnologies.com

 
> 
> 
> Nick Bellinger <nickb@attheoffice.org>
> 16/09/02 09:18
> 
>  
>         To:     Julian Satran/Haifa/IBM@IBMIL
>         cc:     Black_David@emc.com, ips@ece.cmu.edu, cbm@rose.hp.com
>         Subject:        RE: iSCSI: Recognizing Recovery R2Ts
> 
>  
> 
> Julian,
> 
> On Sun, 2002-09-15 at 23:50, Julian Satran wrote:
> > Nick,
> > 
> > I understand that your issue is mainly related to an initiator expecting 
> 
> > "in order" R2Ts and seing an R2T that is out of order.
> > 
> > The simplest answer (and the one that we had in mind when we designed 
> the 
> > protocol) is that in general nodes willing to accept ErrorRecoveryLevels 
> 1 
> > and 2 will probably accept R2Ts out of order. In this case an initiator 
> > does not need to make any distinction.
> > 
> > But not wanting to be that restrictive we let even nodes accepting only 
> > ordered R2Ts do recovery but then (if they care) they may::
> > 
> > initiators:
> > 
> > reject any R2T that is unordered (i.e. do not recover)
> > accept R2T based on having "previously seen it"  (as David Black 
> suggests) 
> > - with the caveat that if the R2T is lost you may have to fall back to 1
> 
> Considering ErrorRecoveryLevel>=1 and DataSequenceinOrder=Yes:
> 
> The potentially lost R2T your referring to is the original one correct? 
> Original meaning the R2T the initiator can use to determine if the R2T
> is infact a recovery R2T and is requesting retransmissing of an
> previously requested offset. 
> 
> If this is the case,  I dont see how this situation could possibly
> happen.  If the R2T the target originally sent gets lost and the
> initiator cannot send any data out without the lost R2T,  how could the
> target possibly be able to generate a recovery r2t with differing
> offsets for a data out that was never received? 
> 
> So if this is indeed correct,  the target generates a sequence timeout
> under the guise of within-command recovery,  and sends a Recovery R2T
> with identical values to the one no data out was received for and
> presumed lost. And then if there is a Data Digest failure on data out
> the target generates a recovery R2T with a previously requested offset,
> and the initiator is able to tell the difference in R2Ts. 
> 
> I think this is a bit confusing (to me anyways) as a recovery R2T can
> either mean:
>                  a) A identical retransmission due to a header digest 
> error (no                   data
> out received, manifests as a sequence timeout)
>                  b) Previously requested offset requested due to a data 
> digest 
> error.
>  
> Is there something I am missing as to how the initiator with
> DataSequenceinOrder=Yes could lose an original R2T and still somehow
> receive a Recovery R2T (of type b) and have within-command recovery
> fail?
> 
> Thanks Julian!
> 
>                  Nicholas Bellinger - Pyxtechnologies.com
> 
> > targets:
> > have only one R2T outstanding if you have both "in order" and error 
> > recovery (a rare thing we believe)
> > 
> > 
> > Regards,
> > Julo
> > 
> > 
> > 
> > 
> > Black_David@emc.com
> > Sent by: owner-ips@ece.cmu.edu
> > 16/09/02 04:58
> > 
> > 
> >         To:     nickb@attheoffice.org
> >         cc:     ips@ece.cmu.edu
> >         Subject:        RE: iSCSI: Recognizing Recovery R2Ts
> > 
> > 
> > 
> > One additional important note - the Recovery R2T generates
> > its own data sequence, so in the following:
> > 
> > > > Following a)
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384 DataSN=2, DataLength=8192
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT F_BIT, Offset=24576, DataSN=3,
> > > > DataLength=8192
> > > > 
> > > > Following B)
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384 DataSN=2, DataLength=8192
> > 
> > The DataSN's should be 0 and 1, not 2 and 3 (and in the B) case above,
> > the F bit needs to be set, as I previously noted).  The initiator really
> > does respond to a Recovery R2T in the same fashion as any other R2T.
> > 
> > Thanks,
> > --David
> > 
> > > -----Original Message-----
> > > From: Black_David@emc.com [mailto:Black_David@emc.com]
> > > Sent: Sunday, September 15, 2002 9:45 PM
> > > To: nickb@attheoffice.org
> > > Cc: ips@ece.cmu.edu
> > > Subject: RE: iSCSI: Recognizing Recovery R2Ts
> > > 
> > > 
> > > > Im still lost here.  Please consider the example:
> > > > 
> > > > MaxDataSegmentLength=8192, MaxBurstLength=32768, ImmediateData=No,
> > > > InitialR2T=Yes.
> > > > 
> > > > I-> ISCSI_INIT_SCSI_CMND, EDTL=131072, SCSI Opcode WRITE_10 
> > > > T-> ISCSI_TARG_R2T, DDTL=32768, Offset=0
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=0, DataSN=0, DataLength=8192
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=8192, DataSN=1, DataLength=8192
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384, DataSN=2, 
> > > DataLength=8192
> > > > 
> > > > (Data Payload Fails on DataSN=2)
> > > > 
> > > > T-> ISCSI_TARG_RJT, Reason=Data Digest (Payload) Error
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, F_BIT, Offset=24576, DataSN=3,
> > > > DataLength=8192
> > > > 
> > > > Can the Target either: (Are both of these legal?)
> > > > 
> > > >              a) Accept DataSN=3 even though DataSN=2 Failed
> > > >                 Only request retransmisson of Failed DataOut
> > > >              T-> ISCSI_TARG_R2T (Recovery) DDTL=8192, Offset=16384
> > > > 
> > > >              or b) Discard DataSN=3, Request retransmission of
> > > >                    all DataOut from failed to MaxBurstLength 
> > > >              T-> ISCSI_TARG_R2T (Recovery) DDTL=16384, Offset=16384
> > > 
> > > Yes, both are legal, a) is preferred for obvious reasons.
> > > 
> > > > Initiator: (This is where im confused)
> > > >              Received either a) or b),  and knows the R2T is a 
> > Recovery
> > > >              R2T because the DDTL is not 32768 (MaxBurstLength or 
> rest 
> > of
> > > >              the original EDTL) and Offset is not 32768.
> > > 
> > > Not exactly.  The R2T is a recovery R2T because it requests data
> > > covered by the previous R2T (Offset < 32768).  This doesn't affect
> > > the initiator's behavior - it just responds to the R2T it receives.
> > > 
> > > > Following a)
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384 DataSN=2, DataLength=8192
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT F_BIT, Offset=24576, DataSN=3,
> > > > DataLength=8192
> > > > 
> > > > Following B)
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384 DataSN=2, DataLength=8192
> > > 
> > > I think you flipped the a) and b) cases above, so you clearly are
> > > confused :-).  In both cases, the initiator looks at the R2T to
> > > determine what data the Target is asking for and sends it - a) asked
> > > for 8k, b) asked for 16k.  The one PDU that responds to a) needs
> > > to have the F bit set, as it completes the response to the second
> > > R2T.  If the target needs to distinguish the recovery data transfer
> > > from the original transfer, it should use a different Target Transfer
> > > Tag in the recovery R2T.
> > > 
> > > > Target:
> > > >              MaxBurstLength is Completed:
> > > > T-> ISCSI_TARG_R2T, Offset=32768, DDTL=32768
> > > > 
> > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=32768, DataSN=0, 
> > > DataLength=8192
> > > 
> > > Ok.
> > > 
> > > > Your statement "Whether the initiator knows this depends on 
> > > whether it
> > > > saw the previous R2T" is still confusing to me.  The logic 
> > > used in the
> > > > above example (EDTL and Offset) I assume is incorrect. 
> > > Would you mind
> > > > elaborating with a specific example of the proper method of 
> > > doing so?
> > > 
> > > I don't understand the problem here.  In all cases, the initiator
> > > looks at the offset and length in the R2T it receives and sends the
> > > data requested by that R2T.  In both cases a) and b) above, 
> > > the initiator
> > > can tell that it's a recovery R2T because the requested data 
> > > range overlaps
> > > a previous R2T.  The statement I wrote covers a completely different
> > > case in which the target sends an R2T that doesn't arrive 
> > > (e.g., header
> > > digest error), and decides to try to recover from that.
> > > 
> > > I also don't understand the reference to EDTL in the command
> > > (as opposed to DDTL in the R2T0.  If a Recovery R2T causes
> > > more than EDTL of data to be transferred for a command that's fine -
> > > the target is in charge of data flow and will tell the initiator when
> > > the command was complete and how much was actually transferred
> > > (consider unexpected end-of-tape on a tape device to understand
> > > why this is done this way).
> > > 
> > > Thanks,
> > > --David
> > > ---------------------------------------------------
> > > David L. Black, Senior Technologist
> > > EMC Corporation, 42 South St., Hopkinton, MA  01748
> > > +1 (508) 249-6449            FAX: +1 (508) 497-8018
> > > black_david@emc.com       Mobile: +1 (978) 394-7754
> > > ---------------------------------------------------
> > > 
> > 
> > 
> 
> 
> 
>