blaze/chest

Prefetch opens too many files

Closed this issue · 0 comments

For a reasonably large chest c, c.prefetch(list(c.keys())) will cause OSError: [Errno 24] Too many open files because open_many opens everything, consumes everything, and then closes everything.

Simple solution: chunk up prefetches into open_many's of some maximum size.