1997 DARPA Research Summary

Defined the storage interface for Network-Attached Secure Disks (NASD). Broadly, network-attached secure disk interfaces should exhibit the following five properties. (1) Direct transfer: Data accessed by a filesystem client is transferred between the NASD drive and the client without indirection (store-and-forward) through a file server machine. (2) Asynchronous filesystem oversight: Frequently consulted but infrequently changed filesystem policy decisions, such as what operations a particular client can perform on a particular set of stored data, are made once by the file manager and asynchronously enforced by the NASD drive. (3) Cryptographic support for request integrity: NASD drives must be capable of computing keyed digests on command and status fields without significant bandwidth penalty. Additionally, NASD drives should offer cryptographic support for high-performance data integrity, data privacy and command privacy. (4) Storage self-management: NASD drives should have knowledge of logical relationships between storage units and exploit these for transparent performance optimization and reliability assurance. (5) Extensibility: NASD drives should be capable of offering extensions for specific, demanding client applications without requiring modification of file manager software.

Specifically, we have defined an interface for NASD drives based on variable-length data objects with exposed attributes describing size, preallocation, event timestamps, clustering, cloning, version, and uninterpreted metadata. NASD objects are organized into dynamically sized partitions each under the control of a single file manager. Objects are created, deleted, named, allocated and copied at the drive. The choice of drives managing variable-length objects instead of storing fixed-length, fixed-location blocks is driven by the need to authorize access to specific data and by the observation that advanced storage systems achieve performance and capacity optimizations by remapping the block address space.

Developed a capability-based NASD security protocol enabling asynchronous verification of outstanding policy decisions. This protocol also allows a NASD drive to be independent of the specifics of its environment's authentication and access control systems. Moreover, the drive's long-term secure state is limited to a secret key hierarchy for each partition. Briefly, a file manager agreeing to a client's access request privately sends to the client a token and a key, which together form an object capability. Client requests on a NASD drive contain the capability token and are digested with the capability key. The drive, using the partition's secret key, constructs the implied capability key as a function of a request's capability token and verifies the request's digest. Agreement of delivered and computed digests ensures the drive that the capability token is authentic and the request's composer holds the corresponding capability key. Revocation exploits the object version and rights expiration time fields of each capability token. For example, immediate revocation is effected by directly modifying the object's version attribute on the drive.

Implemented a prototype NASD drive exhibiting all features of our security protocol and most of the NASD object interface. Operating in the Digital UNIX 3.2g kernel of a DEC Alpha workstation, our prototype emulates a native NASD object store using a modified version of the local UNIX filesystem backed by an HP 2247 1GB disk. Clients and file managers communicate with the drive using DCE 1.0.3 RPC extended by NASD-specific crypto-marshalling and transmitted over a DEC OC3 ATM Gigaswitch. The prototype security implementation uses the reference implementations of the HMAC-MD5 keyed hash from AT&T.

Version 3 of the NFS distributed filesystem has been adapted to use our prototype NASD interface. Encoding NASD drive and object naming in NFS' file handle and piggybacking capability acquisition on NFS pathname component lookup operations, modifications to NFS client code are simple and localized. File read, file write and attribute read operations are redirected directly to the NASD drive. Our initial implementation's performance is within 10% of the native NFS for a single run of the Andrew benchmark and scales the number of active clients and drives much more effectively than native NFS because NASD-NFS file manager load is much less than native NFS. Notably, adding new clients and corresponding NASD storage has no impact on client elapsed time for the available small scale test systems (up to 4 client-drive pairs).

Version 3.4 of the AFS distributed filesystem has also been adapted to use our NASD prototype. Encoding is similar to NFS on NASD, but AFS allows clients to parse directories directly, so new file manager interfaces were added to explicitly obtain capabilities. AFS's cache consistency and quota features make it a more interesting implementation; client writes are batched and each batch obtains a capability immediately before writing and retires that capability immediately after writing. By retiring a write capability before continuing the application, AFS ensures that the file manager can break callbacks on other caching clients. Our AFS implementation uses outstanding capabilities in a similar manner for escrowing quota into writable objects. Initial performance, as for our NFS port, is within 10% of the elapsed time of native AFS for the Andrew benchmark and delivers scalable bandwidth without increase in client elapsed time as the number of active client-drive pairs increase (up to 4 in these tests).

Developed a user-level NFS client able to issue large data accesses into files striped over multiple disks (not supported by NFS or AFS). Rather than modify the NFS server to understand disk striping, we used the NASD-NFS file manager built for the basic NFS port to NASD and added a logical object manager. Clients obtaining a capability from the NFS-NASD file manager will send their first access to the logical object manager which will return a map of the striped object's containing NASDs and the appropriate list of capabilities. Later accesses bypass both NASD-NFS file manager and logical object manager. In scalability tests in which large read access are issued at random offsets in large files, striped NASD-NFS clients achieved linear aggregate bandwidth on up to four client-drive pairs while native NFS on striped (logical volume manager) disks saturates at 66% of the striped NASD-NFS bandwidth.

Developed a storage-embedded informed prefetching system whose collaboration with I/O-intensive applications is independent of and transparent to the NASD-NFS file manager owning the prefetching storage. Demonstrated 40% reduction in elapsed time for out-of-core planar visualization of 3D scientific data whose data is striped over three drives managed by a NASD storage subsystem extended with informed prefetching. The ability to extend intelligent drives for specific applications without modifying operating system and file manager code is essential for a nimble, customer-oriented storage marketplace.

Developed a reservation protocol for a decentralized, scalable video server using internet RSVP/RTP/RTCP protocols. Implemented reservation system in decentralized video server and adapted the Informedia digital library client interface running under Windows/NT to use the reservation protocol.

Negotiated a non-exclusive intellectual property rights sharing agreement between CMU, IBM, HP, Seagate, Quantum creating a working group in the National Storage Industry Consortium (NSIC) that shares pre-standards research results on network-attached storage device systems. This group represents the majority of the disk drive industry and a significant fraction of the storage subsystem marketplace; standards recommendations arising from this group will be influential.