iSCSI: "Wedge" drivers

[Black_David@emc.com]

I guess I need to take my co-chair hat off and say
a few things about EMC's and other "Wedge" drivers.

Wedge drivers are motivated by fault tolerance and
fail-over in addition to load balancing/spreading.
EMC actually has two products, PowerPath for Symmetrix
(does all three), and ATF for Clariion (fail-over only).
Other products include HDS's SafePath, Veritas's DMP
and HP's PV Links.  Apologies to those I've left out.
John's goal of eliminating of wedge drivers via
iSCSI sessions may not be achievable in practice
due to fault tolerance and fail-over concerns.

The basic fault tolerance/failover concern
arises from the requirement that failure of one
interface processor in the disk array should
not disable all access to the host's storage.
Hence the host to storage connectivity usually
encompasses more than one interface processor,
facing array designers with the choice of
using multiple SCSI connections and keeping
the SCSI state in each interface processor
vs. sharing the SCSI state across multiple
interface processors, and making sure everything
works right when one of them fails in an arbitrary
fashion (ouch).  The former is considerably
easier to implement, and requires a wedge driver
on the host side.  The corresponding issue of
how a host deals with possible failure of an
iSCSI HBA has already been noted, and I agree
that the most likely approach is a wedge driver.

Since John works for IBM, I should note that
the phrase "interface processor(s)" isn't directly
applicable to the IBM Shark array, but nonetheless,
this issue is still present -- for fault tolerance,
storage access has to be spread across both RS/6000
systems running AIX in a Shark, and sharing SCSI
state across AIX instances doesn't sound like an
easy thing to do.

The bottom line is that failure and fault tolerance
concerns make it unlikely (IMHO) that an iSCSI session
concept will lead to the extinction of wedge drivers.

There have been some questions about difficulty of
development of wedge drivers.  EMC's experience
is that fault tolerance and fail-over require much
more effort than load balancing for a couple of
reasons.  The first is that more complex systems tend
to have more complex ways of failing than of working
correctly :-(.  The second is that correct behavior
of a wedge driver depends on the correct behavior of
HBA drivers in error cases, and HBA vendors in
general do not exhaustively test correct behavior
of error cases before releasing drivers -- we
find driver bugs on a regular basis.  Given the
intention to implement much/all of iSCSI in
hardware, I suspect that this situation will
continue relatively unchanged.  I'm not sure what
the latter implies for difficulty of development
of error handling code for multi-connection iSCSI
sessions when a single vendor is responsible for
both the hardware and the driver.

--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909
black_david@emc.com  Cellular: +1 (978) 394-7754
---------------------------------------------------