Limit the number of simultaneous batch chunk reqs
UmanShahzad opened this issue · 0 comments
UmanShahzad commented
When we send a batch request for X amount of IPs, and X > 1000, we start chunking X into groups of 1000, and make requests for each group separately to the IPinfo batch endpoint, because the endpoint does not accept a size greater than 1000.
However, we do this all at once for all chunks: if there were 500k IPs as input, 500 simultaneous, parallel requests would be made. The issue with this is we may exceed the file descriptor limit of the user; we had this issue with e.g. the CLI.
Implement an upper-limit on the number of simultaneous, parallel requests being made.