Minio/Flask: payload from `unlinked` very large
cklamann opened this issue · 1 comments
cklamann commented
As the number of files in Minio grows, the number of results in the unlinked
payload is getting very large and in some cases timing out. We'll need to think of a way to load the results in chunks using some kind of server-side filtering.
hannahle commented
There are a couple potential solutions:
- If the user knows the name of the linked file a priori (perhaps by checking Minio or receiving it from someone else), we can perform a validator query versus a full search for all potential buckets and files from Minio.
- Use webhooks to post a new record to the database a file is created (and remove when destroyed) -- then we just query the db using regular fuzzy searches and pagination
I've emailed Hannah Driver so we get more data points on her day-to-day use to choose the appropriate solution.