Too many open files

Question

Too many open files

CardosoJr opened this issue 5 years ago · 3 comments

Hello,

I've been using d6tflow to run multiple ML models with different parameters. In some executions I trained over 20K models.

I'm using 10 workers in a linux environment and after some time I get an error of two many open files. I do not open any file in the custom code inside my tasks. I'm using TaskPqPandas and TaskPickle targets to this for me....

Any help on this issue would be appreciated.

Thanks!

Answer 1 · 2019-11-14T14:36:08.000Z

The problem was at the PickleTarget. There're calls to the open method without the with clause

Answer 2 · 2019-11-17T23:48:58.000Z

Thanks for flagging. Fixed in v0.1.8 can you install dev version and check it works?

Answer 3 · 2020-04-03T01:41:35.000Z

haven't heard back, assume it's fixed, reopen if not