Thread pool size
Closed this issue · 1 comments
I suppose this isn't quite actionable enough, but it might be worth doing some experimentation wrt. the thread pool size. By default rayon uses the CPU count, which isn't necessarily great for IO. In some cases like fast SSDs or high-throughput, high-latency (network) mounts, it might be better to use more threads. The fact that you're also using rayon for some file set operations muddles things up, but probably doesn't matter that much.
This can already be configured with RAYON_NUM_THREADS, but I just wanted to raise the issue here.
I tested varying RAYON_NUM_THREADS, and it seems like the default thread pool size gives the best performance for current local file systems.
Closing for now, may come back to this when the scope of this project expands.