Functionality or example to handle large requests
elbeejay opened this issue · 1 comments
I think that the problem encountered in #64 is likely somewhat common.
There are 2 ways (in my view) that we could help users make larger queries:
-
Create functionality to split up or "chunk" large requests and submit separate queries either in series or parallel to the web service and then return one large merged data frame. This would be handy for the user, but at the cost of increasing the complexity of what is intended to be a fairly thin and simple package.
-
We can create a few examples of how one might write some code to do the "chunking" of large queries themselves, and wrap existing
dataretrieval
functions within a for-loop or similar. This documentation could be referenced in the 414 error message we return, and would not make the package itself any more complex. This option keeps the package light-weight and simple by offloading the complexity associated with large queries to the user.
Thoughts?
My preference is 2. Give a generic looping example in the warning message, then leave it to the user to make it work for their particular application.
In my experience, constructing a loop isn't too hard, and I'd rather not bog you down in documenting the limits of each service and writing one-off examples for each given that the service will be changing.