Manage concurrency limits inside cozy-connector-libs
Opened this issue · 1 comments
I was recommended not to use Promise.all directly too much because we could hit system's limit on open file descriptors.
The user solution is to use Bluebird's Promise.map
with the concurrency
parameter.
However, most connectors will only use our functions like requestFactory
or saveFiles
.
It would probably be a good idea to deal with the concurrency internally at least for the functions we manage. Doing so, most developers won't have to even be aware of our system's limits and of the complexity induced.
For requestFactory
we could add { pool: {maxSockets: Infinity} }
in the default options. Attention, this requires strictSSL
to be at true
(it is at false
today).
For saveFiles
we already use Bluebird and only have to add the concurrency
parameter.
This stack overflow post seems to be a good resource about the maxSockets
option which I totally missed. And also the official nodejs documentation of course.
I suppose I had some connection error because of an old version of nodejs at that time.
I suppose we could then use Promise.all in most cases (even in saveFiles) and avoid the use of bluebird.