how to itterate over a list of urls
Closed this issue · 4 comments
I'm kind of new to cli js programming, and often having trouble fitting what I want to do into such asynchronous models. say I have a list of 100 urls and want to fetch all of them... whats the most elegant and efficient way to do this with the tor.request(success() model? just a loop iteration starts way too many simultaneous requests at once. doing some kind of recursive/stack processing call (i.e. when in success redo request, pop next url until no more urls) seems like it would be very memory inefficient - all the duplicated functions(if not static) and dangling response payloads/bodies. I've also tried(unsuccessfully) to use async, await and promise to cause synchronous execution. not only was I not able to get it to work, but it seemed like a lot of extra code. whats the right(or a good) way to do what I'm talking about (prevent firing so many requests at once / somewhat synonymous processing that is memory efficient). thanks
this looks promising https://stackoverflow.com/questions/8775262/synchronous-requests-in-node-js
negative, it's like nothing is actually awaited
This is actually a very common need/pattern when working with async code. Look at the async npm library -- it is full with all kinds of common async utilities. You'll probably want to use async.series or async.parallelLimit.
async.series: https://caolan.github.io/async/docs.html#series
async.parallelLimit: https://caolan.github.io/async/docs.html#parallelLimit
and the npm package is at: https://www.npmjs.com/package/async