nextest-rs/nextest

v0.9.70 using significantly more stack

jayvdb opened this issue · 7 comments

jayvdb commented

We were using v0.9.68 with prebuilt tests running with RUST_MIN_STACK=80000 in one of our low level libs, as this forced our unit tests to also be proving that they didnt consume a lot of the stack, allowing more stack room for users of these libs.

With https://github.com/nextest-rs/nextest/releases/tag/cargo-nextest-0.9.70 , this limit was suddenly exceeded, and even RUST_MIN_STACK=100000 also failed. One of our libs under this test rig failed even when there are no tests for that crate.

The error we see is

[cargo-make] INFO - Execute Command: "rustup" "run" "stable" "cargo" "nextest" "run" "--all-targets" "--all-features" "--no-fail-fast" "--"
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/app/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc - --crate-name ___ --print=file-names --target x86_64-unknown-linux-gnu --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (signal: 11, SIGSEGV: invalid memory reference)
  --- stderr
  error: rustc interrupted by SIGSEGV, printing backtrace

  /app/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/librustc_driver-cf038663a889d84a.so(+0x2bbe886)[0x7fa0c49b7886]
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7fa0c1cab140]
  /app/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/librustc_driver-cf038663a889d84a.so(+0x4ae501b)[0x7fa0c68de01b]
  /app/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/librustc_driver-cf038663a889d84a.so(+0x4d33df1)[0x7fa0c6b2cdf1]
[cargo-make] ERROR - Error while executing command, exit code: 102
[cargo-make] WARN - Build Failed.
  /app/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/librustc_driver-cf038663a889d84a.so(+0x4d33c4f)[0x7fa0c6b2cc4f]
  /app/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/libstd-2d08990d644ac786.so(rust_metadata_std_85008a29cc0b7dba+0xbf675)[0x7fa0c1d91675]
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7fa0c1c9fea7]
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fa0c1bbfa6f]

  note: we would appreciate a report at https://github.com/rust-lang/rust
Error: Process completed with exit code 1.

The tests pass again when running with RUST_MIN_STACK=140000 , and when using RUST_MIN_STACK=120000 the command cargo nextest run ... doesnt fail as above, but instead it hangs for 30m before I killed it. (edit: nope 130000 also failed; needed to use 140000)

Has there been any recent improvements that run tests with greater use of the stack?

sunshowers commented

Thanks for the report. Stack size is generally not something we've paid much attention to, and there have been a few changes in the area — but I actually think that error is coming from cargo and rustc. I believe the possibility relevant change here is that we now ask cargo for some target-specific information.

Is the challenge that you'd like to set an environment variable for tests? There is some support for that where nextest can do that—for now you can use a setup script. https://nexte.st/book/setup-scripts

jayvdb commented

Ah yes, I saw that enhancement in the commits since v0.9.68 . Any chance that we can provide some envvar, CLI arg or config which would avoid that code path which is invoking cargo ?

Our requirement is that we must be able to verify the maximum stack used by our libs , and we do that by showing all of the unit tests for the libs run within a limited stack size. And we know that in addition, some of the constrained stack size is consumed by the test running, so we have proved that the lib must have some head room as well within that stack limit.

Setting RUST_MIN_STACK is just one way to achieve this.

sunshowers commented

Would it be worth exploring the setup script option? You can set the env for just your test processes, not nextest/cargo/rustc.

sunshowers commented

Actually a target runner might even be better: https://nexte.st/book/target-runners

You can use a shell or batch script that sets the variable.

jayvdb commented

re scripts, will those environment variables will be set in the script be present in the process that the unit tests are run in?

sunshowers commented

re scripts, will those environment variables will be set in the script be present in the process that the unit tests are run in?

Yes, they will be set for whatever tests the setup script was configured to be triggered against (which can be all()).

jayvdb commented

Using scripts solved my use-case. Thank you!