2012 IEEE International Conference on Cluster Computing (CLUSTER), 24-28 Sept. 2012.
Adam Manzanares*, John Bent†, Meghan Wingate*, and Garth A. Gibson‡
*Los Alamos National Laboratory
†EMC Corporation
‡Carnegie Mellon University
Extracting high data bandwidth and metadata rates from parallel file systems is notoriously difficult. User workloads almost never achieve the performance of synthetic benchmarks. The reason for this is that real-world applications are not as well-aligned, well-tuned, or consistent as are synthetic benchmarks. There are at least three possible ways to address this challenge: modification of the real-world workloads, modification of the underlying parallel file systems, or reorganization of the real-world workloads using transformative middleware. In this paper, we demonstrate that transformative middleware is applicable across a large set of high performance computing workloads and is portable across the three major parallel file systems in use today. We also demonstrate that our transformative middleware layer is capable of improving the write, read, and metadata performance of I/O workloads by up to 150x, 10x, and 17x respectively, on workloads with processor counts of up to 65,536.
FULL PAPER: pdf