solvebio/solvebio-python

Ability to paginate through a dataset query in parallel

Opened this issue · 0 comments

Paginating through datasets can be slow especially for large datasets. Adding a parallel query capability through requests-futures or similar would be useful since you can opt-in to the behavior if you have the dependency installed.

The implementation would need to handle the following (the same way the current client code handles them):

  • Parsing the JSON response
  • Handling error responses
  • Handling 429 rate limits

Ideally it will be transparent to the end-user. They can just iterate through a query as usual but behind the scenes it will pull multiple pages in parallel, with a configurable number of "workers".