Prefetch opens too many files
Closed this issue · 0 comments
maxhutch commented
For a reasonably large chest c
, c.prefetch(list(c.keys()))
will cause OSError: [Errno 24] Too many open files
because open_many
opens everything, consumes everything, and then closes everything.
Simple solution: chunk up prefetches into open_many's of some maximum size.