Time is of the essence in high performance computing (HPC) applications, which simulate real-world phenomena impacting our daily lives. Scientific simulations can typically take hours, days, or even weeks to compute. Delays in providing the results of these simulations due to input/output (I/O) slowdowns or failures can have catastrophic impacts.

I/O is the dynamic communication (reading and writing) between the supercomputer, where data is computed, and the file system, where data is stored in files. The larger the scale of the HPC application run, the greater the challenges for I/O. The current trend is that computational capabilities of large-scale supercomputers increase more quickly than I/O capabilities, so the I/O performance promises to get worse instead of better with each generation of supercomputers.

The UnifyFS file system enables HPC science applications to perform I/O operations many times faster than they could with traditional methods. UnifyFS is an ephemeral (or temporary) file system that utilizes fast storage tiers on supercomputers to quickly store and access application data so that applications can produce their results in less time.

Breaking the Bottlenecks

Two primary types of I/O bottlenecks plague HPC applications. When a large number of parallel processes all submit I/O requests to the parallel file system at the same time, I/O slows down—a situation called contention, which is akin to a traffic jam where there are more requests for resources than are available.

In addition, parallel file systems strictly adhere to the semantics of the I/O functions in the Portable Operating System Interface (POSIX), an interface designed decades ago, and not for today’s highly concurrent, parallel workloads. Strict adherence to semantics imposed on the I/O operations in the POSIX interface often require read and write “locks” unnecessary for HPC applications. The overhead of supporting POSIX semantics—effectively locking a file every time it’s accessed—adds unnecessary delay in the completion of I/O operations.

UnifyFS eliminates contention by utilizing fast, node-local storage that is only used by a single application at a time, and it removes unnecessary overheads by relaxing POSIX I/O semantics. UnifyFS is a temporary, virtual file system on the compute node, which means data has less distance to travel and circumvents POSIX obstacles while the application runs. It also eliminates the need for time-consuming development to optimize every application for better I/O performance; UnifyFS itself is the optimization.

Supercomputers are composed of hundreds to thousands of compute nodes (blue rectangles) attached by a high-speed network (blue lines). Top: To access compute node-local storage (cylinders) without UnifyFS, users would need to write code that tracks and determines which other compute node hosts the data it needs, then traverses the network to access the storage device closest to the target compute node (i.e., node-local; arrows) to retrieve data. Bottom: UnifyFS simplifies access to the node-local storage of all compute nodes used by the application. Each process simply performs its I/O operations to the /unifyfs location, where UnifyFS manages all the complexity transparently and with high performance.

Fully Featured

UnifyFS logo of three cylinders surrounded by six circles containing text: user-level installation & operation; production readiness & portability; integration with system resource managers; relaxation of POSIX rules; optimized write operations; automatic file management

UnifyFS provides important features for enabling HPC applications to use node-local storage effectively:

  • It installs and operates at the user level, which means users do not have to ask for system administrator support to install or run it.
  • It is production ready and highly portable from one type of HPC system to another.
  • Its integration with system resource managers ensures easy start up and tear down.
  • It is tailored for HPC I/O behavior and relaxes POSIX rules.
  • Its highly optimized write operations significantly speed up applications.
  • It automatically and efficiently moves files into and out of the file system.

Ultimately, UnifyFS makes I/O fast and easy so users can focus on their science.

Citation

M. J. Brim, A. T. Moody, S. Lim, R. Miller, S. Boehm, C. Stanavige, K. Mohror, S. Oral. 2023. “UnifyFS: A User-level Shared File System for Unified Access to Distributed Local Storage,” In Proceedings of the 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Best Open Source Software Award Winner. ieeexplore.ieee.org/document/10177390