NoPFS is the Near-optimal Prefetching System, an I/O middleware that optimizes distributed machine learning I/O at scale.
For more details, check our paper from Supercomputing 2021: Clairvoyant Prefetching for Distributed Machine Learning I/O.
@inproceedings{dryden2021clairvoyant,
title={CLairvoyant Prefetching for Distributed Machine Learning {I/O}},
author={Nikoli Dryden and Roman B\"{o}hringer and Tal Ben-Nun and Torsten Hoefler},
booktitle={Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC)},
year={2021},
eprint={2101.08734},
archivePrefix={arXiv},
primaryClass={cs.DC}
}
We are currently preparing a more "production-ready" release, which will be out soon! If you are interested in the original "research" code, look at the old-nopfs branch.