Limit the size of pk__in lists when fetching related objects

Question

Limit the size of pk__in lists when fetching related objects

Opened this issue 6 years ago · 2 comments

When fetching related objects by ID in fetch_related we do not currently place any bounds on the the number of IDs to fetch at once (https://github.com/closeio/flask-common/blob/master/flask_common/documents.py#L291). This can result in MongoDB queries with 800 - 1000 IDs in one $in list (pk__in in MongoEngine). Executing such queries with more than 100 - 200 IDs in a single query can result in considerable CPU on the mongod node that's handling the query. Maybe we could execute this query in batches of 100 - 200 IDs at a time (ideally 100) and then combine the results into a single list.

Answer 1 · 2018-06-26T22:38:04.000Z

@tsx I believe this captures what we discussed?

Answer 2 · 2018-06-27T06:03:58.000Z

Yep. See also the issue in the main repo ^