Berkeley Lab's AIIO system diagnoses I/O performance bottlenecks up to 146x faster in HPC applications
Researchers at Lawrence Berkeley National Laboratory used AI to automatically identify I/O bottlenecks at the job level, achieving performance improvements of 1.8x, 2.1x, and 146x across three real scientific applications — without requiring manual expert diagnosis.
Background
HPC systems running large-scale scientific simulations and data-intensive analyses are slowed significantly by inefficient I/O management. Identifying the root cause of an I/O bottleneck in a specific application requires deep expertise and is time-consuming when done manually. Berkeley Lab researchers set out to automate this process at the individual job level — a finer granularity than any prior automated approach.
What Was Implemented
- Developed AIIO, an AI-based system for automatic, job-level I/O performance bottleneck diagnosis
- Performance function selects among five AI model types (MLP, XGBoost, LightGBM, CatBoost, TabNet) per job and domain
- Diagnosis function applies SHAP (SHapley Additive exPlanations) to identify the most impactful performance factors and recommend corrections
- Evaluated on 40 months of Darshan I/O logs from NERSC's Cori system and on six I/O patterns across three active DOE applications: E2E, OpenPMD, and DASSA
- Published peer-reviewed findings at HPDC '23 (August 2023)
Results
Testing AIIO on three real scientific applications produced performance improvements of 1.8x for E2E, 2.1x for OpenPMD, and a remarkable 146x for DASSA. The DASSA result reflects a case where a single identified bottleneck, once addressed, reduced runtime by two orders of magnitude. All improvements were achieved without requiring manual expert diagnosis of the I/O stack.
Lessons
- AI interpretation tools (specifically SHAP) can make machine-learning predictions actionable by revealing which I/O parameters drive poor performance in a specific job
- Job-level granularity is essential: platform-level diagnostics cannot produce recommendations that apply to individual applications
- Multi-model selection (choosing the best among MLP, XGBoost, LightGBM, CatBoost, TabNet per job type) improves accuracy over single-model approaches
- The same framework may enable future runtime systems to self-diagnose and self-correct I/O bottlenecks without human intervention