Presentation

· Contributors · Organizations · Search Program · My Schedule · Happening Now · Maps

This content is available for: Tech Program Reg Pass, Exhibits Reg Pass. Upgrade Registration

Why Wait!? Hades: An Active, Content-Aware System for Precalculating Derived Quantities

DescriptionModern scientific applications produce vast amounts of data, typically stored in monolithic files on parallel file systems (PFS). Analyzing these large files often results in inefficiency due to I/O stalls. To mitigate these stalls, certain data can be pre-computed during the production phase and queried during analysis. However, this solution demands added storage capacity and an astute use of storage hierarchies. In this context, we introduce Hades, an I/O engine seamlessly integrated with the Adios2 framework. Hades offers hierarchical buffering, which enables smart data placement and prefetching across the spectrum of I/O devices. Additionally, it is adept at computing basic derived quantities required by I/O applications, such as the global and local min/max values. A notable feature of Hades is its memory-first metadata management strategy, which is designed for querying derived data, significantly enhancing system performance.

Authors

Jaime Cernuda

Illinois Institute of Technology

Luke Logan

Illinois Institute of Technology