Accelerate upload of a bunch of small files - targz upload
pierreadorni opened this issue · 1 comments
The issue minio/minio#8715 and the subsequent PR minio/minio#11849 added a more effective way to upload a large number of small files, by decoding a targz stream. I think it would be nice to integrate this feature in the console !
Expected Behavior
Provide an option to upload a folder as a targz stream, or change the behavior of the "upload folder" action.
Current Behavior
Currently, the "upload folder" option in the UI seems to upload each file in the folder separately, which adds quite a bit of overhead for a large numbers of small files
Possible Solution
I'm not sure if this is easily doable in a SPA, but maybe a library like tar-stream could help
Context
I stumbled upon this issue while using minio for storage of large computer vision datasets for deep learning applications: an upload of thousands of small images is a really long process when the files are sent one by one.
Regression
Not a regression
Your Environment
- MinIO version used (
minio --version): RELEASE.2024-03-26T22-10-45Z (go1.21.8 linux/amd64) - Server setup and configuration: run in Docker
- Operating System and version (
uname -a): windows 11
I stumbled upon this issue while using minio for storage of large computer vision datasets for deep learning applications: an upload of thousands of small images is a really long process when the files are sent one by one.
Do not build critical infrastructure in silo's buy a subscription to get your setup, architecture reviewed.