Re: iSCSI Data Integrity - Digests

[Sean Quinlan <seanq@research.bell-labs.com>]

Glen,

Glen Turner wrote:
> 
> Sean Quinlan wrote:
> 
> > If TLS or IPsec are used in conjunction with iSCSI, then strong
> > end-to-end integrity is already provided and duplicating this in iSCSI seem
> > rather a waste.
> 
> Um, this doesn't match the paper presented at SIGCOMM2000 showing an
> uncomfortably high level of errors *above* the TCP layer due to faulty
> computer hardware. [1]
> 
> The paper strongly recommended application-layer checksums.

I think the paper you mention is
	"When the CRC and TCP checksum disagree" by Stone and Partridge
http://www.acm.org/sigcomm/sigcomm2000/conf/techprog.htm

I had not seen this paper and it is certainly an interesting read.
One of the conclusions of the paper is that if you really care about
your data you should do a checksum before the data has passed through a
DMA engine.  They mention SSL (also know as TLS) as one way to do this, but
claim that IPsec is not a good option since it is often done by hardware
assist in the NIC after the DMA - Then again if you really believe this, then
you need to do checksums before passing data to your plain old SCSI card, since
this also uses a DMA engine.  This seems a little impractical! Also, considering much
of the iSCSI discussions appears to expect that iSCSI will be offloaded onto
hardware on an adapter card, a checksum at the iSCSI layer would probably
occur after the DMA engine as well...  The paper states
	In the past, one of the authors has periodically recommended
	improving transmission performance by doing the checksum as part of
	the DMA process or in the transmission path in the network interface.
	Base on this study, we can now say that advice is wrong because it
	leave data too exposed to hardware errors.
This sort of goes against the current trend in high speed NICs which provide
offloading of TCP checksums...  My guess is people want the speed provided
such offloading and if the DMA is really causing high error rates, then
the solution is to build more reliable DMA/PCI interfaces.

As a side note, I had a look at the number of TCP checksum errors on
out outside HTTP server and on an internal cpu server.  The outside
machine an error rate of about 1 in 15,000 while the internal machine had
an error rate of 1 in 1,000,000.  Both these numbers where higher than
I would have expected!  Thus, the paper certainly has a point!

> 
> Also, your analogy with HTTP is a bit misleading.  iSCSI errors
> are cumulative, as disks are usually attached to iSCSI sessions
> and a write corruption is stored to disk for later reading.
> 

HTTP might not be a perfect analogy, but I believe NFS and CIFS are pretty
close.  I believe almost every byte written to a NetApp server goes
over one of these two protocols and does not include more integrity checking than
provided by TCP and the link layer.

> > If data integrity is provided by iSCSI, CRC algorithms are not particularly
> > ideal from a software implementation point of view - surely there
> > are better alternatives than CRC that are easy to implement in hardware and
> > can take advantage of 32bit wide data ops.
> 
> There is unlikely to be an algorithm better than CRC that is also
> faster than CRC.  Essentially, to handle data in words it appears
> that you need to trade off the level of protection.  I know of no
> paper that explores this assumption, but math is not my strong point.
> 

I am certainly no expert in checksums, but I believe the adler32 checksum
used in zip (rfc1950) and SCTP (rfc2960) is claimed to be as good as a 32 bit
CRC and it is more friendly to software implementation on a 32 bit
machine, though given that it uses the mod operator it is more difficult
than CRC to implement in hardware.  Secure hashes such as md5 are also
fairly reasonable in software with the added bonus that they can resist
tampering - these are also probably difficult to do in hardware.