RE: iSCSI Target Reset

[Roger Cummings <roger.cummings@veritas.com>]

Sandeep,

Your statements about VERITAS Cluster Server (VCS) are correct. Today VCS
supports 32 node clusters and uses SCSI reservations in its storage
allocation and split brain detection algorithms. Target resets are also part
of the split brain algorithm, though they are not used in any major way. For
future configurations we're looking at using Persistent Reservations, as
defined in SPC-2 and beyond, for many of the reasons already stated in this
thread i.e. scalability and data availability in large SANs with multiported
storage.

Resetting only the LUNs that a specific initiator can see, or providing a
LUN-specific reset, is an interesting idea, and I think there are a number
of storage controllers out there that provide such functionality in a vendor
unique way. But my personal feeling (not necessarily that of everyone @
VERITAS) is that using & extending the functionality provided by Persistent
Reservations is a better and cleaner solution going forward.

Regards,





Roger Cummings
Technology Group
VERITAS Software

roger.cummings@veritas.com

-----Original Message-----
From: Sandeep Joshi [mailto:sandeepj@research.bell-labs.com]
Sent: Tuesday, April 24, 2001 11:27 AM
To: John Hufferd
Cc: ips@ece.cmu.edu
Subject: Re: iSCSI Target Reset


John Hufferd wrote:
> 
> I am posting Charles comments to me onto the reflector, you all might find
> it interesting.  Thank you Charles.
> 
> Any other comments?

It would be interesting to hear from Veritas (anyone on
the list?).  I think the Veritas Cluster Server also uses 
SCSI reservations to avoid split-brain and can support 
32-node clusters in a SAN.

-Sandeep

> 
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Internet address: hufferd@us.ibm.com
> ---------------------- Forwarded by John Hufferd/San Jose/IBM on
04/24/2001
> 02:37 AM ---------------------------
> 
> Charles Monia <cmonia@NishanSystems.com> on 04/24/2001 12:02:36 AM
> 
> To:   John Hufferd/San Jose/IBM@IBMUS, Charles Monia
>       <cmonia@NishanSystems.com>
> cc:
> Subject:  RE: iSCSI Target Reset
> 
> Hi John;
> 
> The following is my .02:
> 
> 1.  Target reset must be supported (SAM says so at the moment).
> 
> 2.  The interconnect's behavior is outside the scope of SAM.  i.e..  It is
> up to the protocol spec.
> 
> 3.  IMO: The only SAM requirement is the behavior at device (LUN) level as
> seen by the  initiator issuing the request.  In that regard, for example,
> it's sufficient to reset only the LUs that the initiator can see.  In a
> virtual environment I assume that's a small subset of the LUs on a system.
> 
> Charles
> 
> > -----Original Message-----
> > From: John Hufferd [mailto:hufferd@us.ibm.com]
> > Sent: Monday, April 23, 2001 10:43 PM
> > To: Charles Monia
> > Subject: RE: iSCSI Target Reset
> >
> >
> >
> > Charles,
> > You need to be more direct.  Is it a T10 requirement to support Target
> > Reset?  How much Flexibility does T10 give the implementations.
> >
> > .
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/SSG San Jose Ca
> > (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > Internet address: hufferd@us.ibm.com
> >
> >
> > Charles Monia <cmonia@NishanSystems.com>@ece.cmu.edu on
> > 04/23/2001 08:36:20
> > PM
> >
> > Sent by:  owner-ips@ece.cmu.edu
> >
> >
> > To:   ips@ece.cmu.edu
> > cc:   Charles Monia <cmonia@NishanSystems.com>
> > Subject:  RE: iSCSI Target Reset
> >
> >
> >
> > Hi:
> >
> > Sorry to reopen old issues, but unfortunately that seems
> > necessary since
> > something has been lost in translation.
> >
> > My concerns about TARGET RESET center on three areas:
> >
> > 1.  Adverse side effects at the transport layer that could
> > affect other
> > users.
> > 2.  Similar adverse side effects on the affected devices,
> > 3.  The impact on legacy software.
> >
> > The gist of my opinion on the first issue, (as expressed on the T10
> > reflector) is as follows:
> >
> > "> > > > The ... issue is whether mechanisms, such as terget reset,
> > > > > > are appropriate for a given transport.  In my view, the only
> > > > > > immutable requirement is to preserve the transport-independant
> > > > > > part of the semantics.  The definition of transport-specific
> > > > > > side effects is best handled in the appropriate transport
> > > > > > specification."
> >
> > The point of the above is that, in my view, a protocol
> > specification has
> > leeway to define the protocol-specific side effects in a rational and
> > benign
> > manner provided that the observable effects on the attached
> > devices are
> > preserved.
> >
> > Regarding adverse device-level side effects, I also stated that:
> >
> > "....restricting the operation [target reset] means providing
> > hooks so that
> > only a
> > trusted class of initiators can perform the function.  It's a bit like
> > controlling access to a file so that lots of users can read
> > it but only a
> > trusted few can perform a write or delete operation."
> >
> > Finally, with regard to legacy implementations, my main
> > intent was to avoid
> > the situation where an operation that was previously legal
> > becomes illegal.
> > In that regard, I also recall suggesting that the function be
> > treated as a
> > LUN RESET broadcast to all the logical units to which the initator had
> > access privileges.  That would also support the notion of preventing
> > adverse
> > effects on other users as well.
> >
> > At any rate, since there is a proposal to make LUN RESET
> > mandatory (see
> > below), I believed this was a reasonable and consistent alternative. I
> > therefore felt (and continue to believe) there is no justification for
> > making the function optional.
> >
> > Anyhow, thess views did not prevail at the meeting in
> > question. For that
> > reason, the so-called NOP proposal was made as a last ditch effort to
> > accomodate legacy implementations by preserving a measure of backwards
> > compatibility (albeit token compatibility). In that regard,
> > it seemed the
> > lesser evil.
> >
> > > > As for leaving things out of iSCSI - the default modus
> > > > operandi should be to put in everything that's described
> > > > in SAM2 unless we can convince T10 to take the feature
> > > > out of SAM2.  Let's not go deciding to cast things out
> > > > of SCSI on T10's behalf.
> >
> > So, I guess that anyone wishing to support a change in the spec is, of
> > course, free to pursue it in that forum.
> >
> > Charles
> >
> > PS: The proposal referenced above can be found at:
> > ftp://ftp.t10.org/t10/document.01/01-015r2.pdf
> >
> > > -----Original Message-----
> > > From: Elliott, Robert [mailto:Robert.Elliott@COMPAQ.com]
> > > Sent: Monday, April 23, 2001 5:20 PM
> > > To: ips@ece.cmu.edu; 'Black_David@emc.com';
> > 'cmonia@nishansystems.com'
> > > Subject: RE: iSCSI Target Reset
> > >
> > >
> > > It's not a very good situation when each device chooses the
> > > interpretation of TARGET RESET that it thinks is appropriate.
> > > IBM's Shark might choose a different hack than Compaq's
> > > RA8000.  How is software supposed to make sense of this?
> > >
> > > The rule in SAM-2 is that TARGET RESET resets every logical
> > > unit (subject to access controls, if implemented).  The fact
> > > is that in Fibre Channel, not many multi-LUN multi-port
> > > targets followed that rule.  The result is that software
> > > cannot tell what's going to happen and may have to handle
> > > targets from each vendor differently.  This is not very
> > > interoperable.
> > >
> > > Charles suggested in the T10 meeting that we allow it to
> > > be implemented as a no-op rather than let protocols drop
> > > support for it.  That doesn't help software like Windows that
> > > does expect certain effects - a no-op implementation
> > > would break clustering.  By removing it from the protocol,
> > > software is forced to find a suitable replacement (e.g. use
> > > LOGICAL UNIT RESET or switch to persistent reservations).
> > > In Windows, this can be done at the port driver level
> > > (STORPORT improving on SCSIPORT) or at the miniport level
> > > (convert each target reset request into multiple LOGICAL UNIT
> > > RESETs).
> > >
> > > Note that other protocols like NFS and HTTP over IP don't
> > > seem to have "server resets."
> > >
> > > The recent SAM-2 change was expressly designed to encourage
> > > iSCSI and SRP to drop support for TARGET RESET.  Please don't
> > > keep it because you think T10 would be offended :-)
> > >
> > > ---
> > > Rob Elliott, Compaq Server Storage
> > > Robert.Elliott@compaq.com
> > >
> > >
> > >
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Black_David@emc.com [mailto:Black_David@emc.com]
> > > > Sent: Monday, April 23, 2001 6:31 PM
> > > > To: cmonia@NishanSystems.com; ips@ece.cmu.edu
> > > > Subject: RE: iSCSI Target Reset
> > > >
> > > >
> > > > I agree with Charles that this is an implementation
> > > > issue.  If a Shark wants to reset all 32 adapters
> > > > when it receives a Target Reset on one of them, that's
> > > > a Shark implementation decision.  It's completely valid
> > > > to reset only the adapter that the Target Reset is
> > > > received on (common Fibre Channel behavior) or
> > > > only the iSCSI target to which the Target Reset is
> > > > addressed if there's more than one Target behind
> > > > the adapter.
> > > >
> > > > As for leaving things out of iSCSI - the default modus
> > > > operandi should be to put in everything that's described
> > > > in SAM2 unless we can convince T10 to take the feature
> > > > out of SAM2.  Let's not go deciding to cast things out
> > > > of SCSI on T10's behalf.
> > > >
> > > > Thanks,
> > > > --David
> > > >
> > > > > -----Original Message-----
> > > > > From:    Charles Monia [SMTP:cmonia@nishansystems.com]
> > > > > Sent:    Monday, April 23, 2001 7:12 PM
> > > > > To: ips@ece.cmu.edu
> > > > > Subject: RE: iSCSI Target Reset
> > > > >
> > > > > Hi:
> > > > >
> > > > > These seem to be implementation decisions. I don't see how
> > > > that justifies
> > > > > removing support from the protocol.
> > > > >
> > > > > Charles
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: John Hufferd [mailto:hufferd@us.ibm.com]
> > > > > > Sent: Monday, April 23, 2001 2:34 PM
> > > > > > To: Santosh Rao
> > > > > > Cc: ips@ece.cmu.edu
> > > > > > Subject: Re: iSCSI Target Reset
> > > > > >
> > > > > >
> > > > > >
> > > > > > Absolutely not,  Why would we think that impacting 32
> > > > different other
> > > > > > initiators is an OK thing to do.  By the way there are lots
> > > > > > more Initiators
> > > > > > possible with FC on Shark, and would hope that there would be
> > > > > > even more
> > > > > > with iSCSI.
> > > > > >
> > > > > > I have been told that these large Storage Controllers do not
> > > > > > support Target
> > > > > > Reset today.  So I see no loss in not supporting such an
> > > > item in iSCSI
> > > > > > especially since many Initiators will be beyond even the
> > > > distances and
> > > > > > mischief that is possible with FC.
> > > > > >
> > > > > > .
> > > > > > .
> > > > > > .
> > > > > > John L. Hufferd
> > > > > > Senior Technical Staff Member (STSM)
> > > > > > IBM/SSG San Jose Ca
> > > > > > (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > > > > > Internet address: hufferd@us.ibm.com
> > > > > >
> > > > > >
> > > > > > Santosh Rao <santoshr@cup.hp.com>@ece.cmu.edu on 04/23/2001
> > > > > > 01:24:02 PM
> > > > > >
> > > > > > Sent by:  owner-ips@ece.cmu.edu
> > > > > >
> > > > > >
> > > > > > To:   ips@ece.cmu.edu
> > > > > > cc:
> > > > > > Subject:  Re: iSCSI Target Reset
> > > > > >
> > > > > >
> > > > > >
> > > > > > "Dillard, David" wrote:
> > > > > > >
> > > > > > > When will STORPORT be generally available?  The latest
> > > > > > STORPORT document
> > > > > > > that I found on the MS web site is version 0.6a, dated
> > > > > > March 18, 2001.
> > > > > > > Given this it seems like STORPORT might not be available
> > > > > > soon.  In that
> > > > > > case
> > > > > > > do you know what happens with the current drivers?  Are we
> > > > > > going to be
> > > > > > > telling customers that if they want to use iSCSI and NT
> > > > > > clustering they
> > > > > > have
> > > > > > > to update to Whistler?
> > > > > >
> > > > > >
> > > > > > [One would hope that this list does not turn into a Microsoft
> > > > > > release/product discussion mailing list (?) ]
> > > > > >
> > > > > > Without going into specifics of A certain O.S., does it
> > > suffice to
> > > > > > require that iSCSI not break existing legacy SCSI
> > applications ?
> > > > > >
> > > > > > If the above is a valid requirement, then, knowing that legacy
> > > > > > applications continue to use SCSI-2 Reserve/Release and the
> > > > > > target reset
> > > > > > as a mechanism of breaking SCSI-2 reservations, should'nt
> > > > > > iSCSI continue
> > > > > > to support the target reset ?
> > > > > >
> > > > > > - Santosh
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > >
> > >
> >
> >
> >