RE: iSCSI: Immediate Data

["John Hufferd" <hufferd@us.ibm.com>]

Rober, Comment in line.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: hufferd@us.ibm.com


Robert Snively <rsnively@brocade.com> on 10/02/2001 10:06:04 AM

To:   John Hufferd/San Jose/IBM@IBMUS, ips@ece.cmu.edu
cc:
Subject:  RE: iSCSI: Immediate Data



John,

Some minor disagreements here:

> 1. Immediate Data permits the elimination of an additional network
> handshake/interaction.

Immediate Data is not necessary to eliminate the handshake.  Use
Unsolicited Data with InitialR2T = no, and (assuming you have a
decent initiator) you get the same result.
[Huff]
The point is that Immediate Data is simpler then the above since it is with
the CDB
[/Huff]

>
> 2. This is important since some applications have a key sensitivity to
> Latency. The most important of these is Database.

Database is certainly sensitive to latency, but the real
latency is to the SCSI Response, which formalizes the promise
that the data will never, ever, be lost from the storage.
Even the use of InitialR2T = yes on long links is a second-order
performance effect on this if your storage is a disk drive.

[Huff]Now you are firmly on my turf.  Almost all Raid Controllers, have
Caches with Non Volatile Storage.  (Or at least all the ones to which
Database response is important have Non Volatile RAM for Write Caching.)
The Non Volatile Write to Cache is what takes the Disk Latency out of the
equation.  So with immediate data the response to small writes can be very
fast.
[/Huff]

> 3. Not only is Database sensitive to Latency, it generally
> has small I/O
> writes.

This is application dependent.  Most database programs out of the
box tend to use small block transfers, but well-tuned databases
that are doing work other than OLTP tend to cluster writes into
much larger blocks when possible.
[Huff]
Most of the I/O operations are still small.  It is from these that we need
to remove as much latency as possible.
We do not need to do anything special for the Large I/O.
[/Huff]

> 4. The hardware HBAs that many of you folks are building,
> include a TCP/IP
> Offload Engines (TOEs).  This permits support of Gigabit Line
> speed without
> Server Load.  However, even though many of you are attempting to
> parallelize as much of the TOE processing as possible, TCP/IP
> processing
> will still add latency to each iSCSI interaction, as compared to Fibre
> Channel.

What can I say?  You have hit a basic truth.

....  snip   ....

>
> 7. So it is the Immediate Data feature of iSCSI that will
> make an important
> difference in the Key Response Time Sensitive Applications,
> which by luck
> only transfer a small amounts of data at a time.

See 1-3.
[Huff] and my responses.
[/Huff]

> 8. When we attempt to let iSCSI shine on the "at-distance" storage
> environment, the value of Immediate Data, and Unsolicited
> Data are even
> more valuable.  However, my concern at this time is for
> Immediate Data.

I am absolutely in agreement with your assessment of unsolicited
data at distance, but immediate data is not a necessary
part of that improvement.

[Huff] I can not understand why you dislike immediate data, and still seem
to like unsolicited data. You will have almost all the same iSCSI code path
in the node anyway if you handle Login.  Why you would not like a code path
that makes this important reduction, seems a bit strange.
[/Huff]

As for simplicity:

> A. Creating a Buffer Manager that can allocate buffers that
> are chained
> together, is kind of fundamental to the high performance
> environment we are
> attempting to work with.  All the processes (whether iSCSI or
> SCSI) will
> allocate buffers (from the Buffer Manager), place data in one
> or more of
> these buffers, and pass the pointer list to the next process.
> There will be
> no copying of data.

See my previous note considering the actual destination of
data, which is not in a memory address space.  It is in this
way that storage differs from networking.  You can't pass
pointers about, but must place the data into a storage
space that has the properties of non-volatility, redundancy,
coherence, and accessibility through the SCSI command set.

[Huff] I have no idea why you say this.  Our storage controllers and our
competition has been doing this for years.  The Non Volatile (RAM) Memory
is completely addressable.  And yes the Cache Manager looks a lot like all
buffer managers, and there are a lot of pointers.
[/Huff]

...   snip   ...