Merlin SIM fails w/ many files and `memfd`
sk1p opened this issue · 0 comments
sk1p commented
Debugging with @matbryan52: if the Merlin dataset has many files (thousands), it will exhaust the number of file descriptors available by default. Increasing the limit can then cause another issue, as we are using select
to check for connection abortions, and select
only supports file descriptors in [0, 1024)
.
Why we are using select
: Windows. See commit ec9953c
Possible fix is to keep using select
on Windows, and switch to the previous implementation using poll
or epoll
for Linux, which works properly for "high" file descriptor numbers.