Internal storage cannot handle more than a thousand files
Closed this issue · 3 comments
Describe the bug
When uploading more than 999 files, the interface will only show a 1000 files. I've tested this with 2000 files of 10K.
In addition, when downloading the files, only a 1000 of the 2000 files are within the zip file on safari. On firefox the download fails.
If only 950 files are uploaded, the file count is OK in zips downloaded with safari and firefox. Also the download on firefox doesn't fail.
@emielvdveen I will provide you with a to-be-created tool to help you upload files to S3 for testing purposes.
To Reproduce
Steps to reproduce the behavior:
- Upload 1000+ files into the appropriate project location
- Browse to the storage component of the data donation
- Observe that the file count is 1000 even though more files have been uploaded
- Observe that the download on firefox fails
- Observe that the download on safari succeeds but does not contain the correct amount of files.
Expected behavior
- Correct number of files are shown in interface
- Correct number of files are present in succesfull downloaded zip
- Download on firefox succeeds.
Screenshots
Firefox download fails:
https://github.com/eyra/mono/assets/88683839/78367dec-d6fc-4891-9c11-b970f56bb6e5
Desktop (please complete the following information):
- safari
- firefox
Ik denk dat ik weet waarom dit is. S3 doet file listing per page van 1000 en de huidige code vraagt niet door.
@emielvdveen script om te testen zit nu in devops master (git pull) zie ook link op basecamp naar notion pagina voor beetje context
Solved, just dealt with 1100 files perfectly