/dataflux-pytorch

The Cloud Storage Connector for PyTorch is an effort to improve ML-training efficiency when using data stored in GCS for training datasets. Using the Connector for PyTorch for training is up to 3X faster when the dataset consists of many small files (e.g., 100 - 500 KB).

Primary LanguagePythonApache License 2.0Apache-2.0

Watchers