konnectors/libs

Manage concurrency limits inside cozy-connector-libs

Opened this issue · 1 comments

edas commented

I was recommended not to use Promise.all directly too much because we could hit system's limit on open file descriptors.

The user solution is to use Bluebird's Promise.map with the concurrency parameter.

However, most connectors will only use our functions like requestFactory or saveFiles.
It would probably be a good idea to deal with the concurrency internally at least for the functions we manage. Doing so, most developers won't have to even be aware of our system's limits and of the complexity induced.

For requestFactory we could add { pool: {maxSockets: Infinity} } in the default options. Attention, this requires strictSSL to be at true (it is at falsetoday).

For saveFiles we already use Bluebird and only have to add the concurrency parameter.

This stack overflow post seems to be a good resource about the maxSockets option which I totally missed. And also the official nodejs documentation of course.

I suppose I had some connection error because of an old version of nodejs at that time.

I suppose we could then use Promise.all in most cases (even in saveFiles) and avoid the use of bluebird.