PDL Abstract

Runtime Estimation and Resource Allocation for Concurrency Testing

Carnegie Mellon Univsersity Parallel Data Laboratory Technical Report CMU-PDL-12-115, November 2012.

Chuck Cranor, Milo Polte, Garth Gibson

Parallel Data Laboratory
Carnegie Mellon University
Pittsburgh, PA 15213


In this report we describe how we adapted the Parallel Log Structured Filesystem (PLFS) to enable HPC applications to be able read and write data from the HDFS cloud storage subsystem. Our enhanced version of PLFS provides HPC applications with the ability to concurrently write from multiple compute nodes into a single file stored in HDFS, thus allowing HPC applications to checkpoint. Our results show that HDFS combined with our PLFS HDFS I/O Store module is able to handle a concurrent write checkpoint workload generated by a benchmark with good performance.

KEYWORDS: HPC Computing, Cloud Storage, PLFS, HDFS, PVFS