test utils::permuted_graph::test_sorted_permuted_graph fails with "Too many open files"
zacchiro opened this issue · 6 comments
As per issue title. Here's a log from my computer:
$ cargo t sorted_permuted_graph
Finished test [unoptimized + debuginfo] target(s) in 0.03s
Running unittests src/lib.rs (target/debug/deps/webgraph-55ec38795ff86d05)
running 1 test
test utils::permuted_graph::test_sorted_permuted_graph ... FAILED
failures:
---- utils::permuted_graph::test_sorted_permuted_graph stdout ----
thread 'utils::permuted_graph::test_sorted_permuted_graph' panicked at 'called `Result::unwrap()` on an `Err` value:
Os { code: 24, kind: Uncategorized, message: "Too many open files" }', src/utils/sort_pairs.rs:75:81
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Interestingly enough, the test does not fail on the CI (see example from a recent run).
FWIW, the above was with 1024 max file descriptors (according to ulimit).
I've bisected this and it appears that the minimum number of maximum file descriptors needed to make the test pass is around 4100.
That seems a bit much :-)
This part of the test_sorted_permuted_graph opens exactly 4001 files, I reduced it to 401.
for batch_size in vec![1, 10, 100] {
let mut s = Sorted::new(4, batch_size)?;
for _ in 0..1000 {
s.push(1, 2)?;
s.push(2, 2)?;
s.push(2, 1)?;
s.push(1, 1)?;
}Just increase the of allowed open files. The test abuses a bit of that.
Do you mean by explicitly calling ulimit from within the test? (That'd be fine with me.)
Otherwise, expecting users that run the test suite to do that themselves to make the test pass won't work well.
with something like this: https://docs.rs/rlimit/0.9.1/rlimit/fn.increase_nofile_limit.html