ccmbioinfo/stager

Minio/Flask: payload from `unlinked` very large

cklamann opened this issue · 1 comments

As the number of files in Minio grows, the number of results in the unlinked payload is getting very large and in some cases timing out. We'll need to think of a way to load the results in chunks using some kind of server-side filtering.

There are a couple potential solutions:

  1. If the user knows the name of the linked file a priori (perhaps by checking Minio or receiving it from someone else), we can perform a validator query versus a full search for all potential buckets and files from Minio.
  2. Use webhooks to post a new record to the database a file is created (and remove when destroyed) -- then we just query the db using regular fuzzy searches and pagination

I've emailed Hannah Driver so we get more data points on her day-to-day use to choose the appropriate solution.