Presentation
SIGN IN TO VIEW THIS PRESENTATION Sign In
Optimizing Write Performance for Checkpointing to Parallel File Systems Using LSM-Trees
DescriptionThe widening gap between compute and I/O performances on modern HPC systems means that writing checkpoints to a parallel file system for fault tolerance is fast becoming a bottleneck to high-performance. It is therefore vital that software is engineered such that it can achieve the highest proportion of available performance on the underlying hardware; and this is a burden often carried by I/O middleware libraries. In this paper, we outline such an I/O library based on a Log-structured Merge Tree (LSM-Tree), not just for metadata, but also scientific data. We benchmark its performance using the IOR benchmark, demonstrating 2.4 to 76.7x better performance than alternative file formats, such as ADIOS2, HDF5, and IOR baseline when running on a Lustre Parallel File System. We further demonstrate that when our LSM-Tree I/O library is used as a storage-layer for ADIOS2, the resulting I/O library still outperforms the default ADIOS2 implementation by 1.5x.
Event Type
Workshop
TimeSunday, 12 November 20233:30pm - 3:55pm MST
Location605
Fault Handling and Tolerance
Large Scale Systems
W