[Question] Meaning of 'filtered out' in the test result
Closed this issue · 8 comments
GWS0428 commented
After submitting my code to gg, I received the 75/100 (score, HW1). However, upon reviewing the log file, it states 'test result : OK' for all tests. The only explanation I can deduce for receiving a partial score is that the log file sometimes shows 'filered out 3' as follows.
What is the meaning of 'filtered out'?
test cache_no_duplicate_sequential ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
Finished test profile [unoptimized + debuginfo] target(s) in 0.07s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8) ```
Lee-Janggun commented
In the grade script, we run the tests one by one in separate 'cargo test' invocations to give partial grades. '3 filtered out' means the three tests that were not ran in that specific 'cargo test' invocation.
08kmc09 commented
https://doc.rust-lang.org/book/ch11-02-running-tests.html
You may refer to the Rust book for more information about test.
Oh I didn't fully understand about your question. It's quite misleading answer.
Lee-Janggun commented
Although... the fact that it said ok but you didn't get points is a bit weird. Can you post the full log?
GWS0428 commented
Here is my full log
===== SETUP =====
===== CHECK =====
[2024-03-30 09:43:25,497: WARNING/ForkPoolWorker-32] Demoting to runner...
info: downloading installer
info: profile set to 'default'
info: default host triple is x86_64-unknown-linux-gnu
warning: Updating existing toolchain, profile choice will be ignored
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
stable-x86_64-unknown-linux-gnu unchanged - rustc 1.77.1 (7cf61ebde 2024-03-27)
Rust is installed now. Great!
To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).
To configure your current shell, you need to source
the corresponding env file under $HOME/.cargo.
This is usually done by running one of the following (note the leading DOT):
. "$HOME/.cargo/env" # For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish" # For fish
info: component 'rust-src' is up to date
Archive: submission.zip
inflating: submission/cache.rs
inflating: submission/tcp.rs
inflating: submission/thread_pool.rs
Cloning into 'cs431'...
info: syncing channel updates for 'nightly-2024-03-13-x86_64-unknown-linux-gnu'
nightly-2024-03-13-x86_64-unknown-linux-gnu unchanged - rustc 1.78.0-nightly (a165f1f65 2024-03-12)
info: checking for self-update
info: component 'rust-src' is up to date
Updating crates.io index
Updating git repository `https://github.com/kaist-cp/cs431`
Compiling crossbeam-utils v0.8.19
Compiling libc v0.2.153
Checking cfg-if v1.0.0
Compiling rayon-core v1.12.1
Checking memchr v2.7.2
Checking regex-syntax v0.8.3
Checking ppv-lite86 v0.2.17
Checking either v1.10.0
Checking aho-corasick v1.1.3
Checking crossbeam-epoch v0.9.18
Checking crossbeam-channel v0.5.12
Checking crossbeam-deque v0.8.5
Checking cs431 v0.1.0 (https://github.com/kaist-cp/cs431#0bb98050)
Checking getrandom v0.2.12
Checking rand_core v0.6.4
Checking rand_chacha v0.3.1
Checking rayon v1.10.0
Checking rand v0.8.5
Checking regex-automata v0.4.6
Checking regex v1.10.4
Checking cs431-homework v0.1.0 (/app/files/checks/8c1728a7-85fc-404c-9d70-9034ef6460b6/cs431/homework)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5.33s
Running with cargo...
Testing basic functionalities of cache.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
Running tests/cache.rs (target/debug/deps/cache-d94123a79a74e29b)
running 1 test
test cache_no_duplicate_sequential ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
Running tests/cache.rs (target/debug/deps/cache-d94123a79a74e29b)
running 1 test
test cache_no_duplicate_concurrent ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.02s
Testing nonblockingness of cache.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
Running tests/cache.rs (target/debug/deps/cache-d94123a79a74e29b)
running 1 test
test cache_no_block_disjoint ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
Running tests/cache.rs (target/debug/deps/cache-d94123a79a74e29b)
running 1 test
Test timed out: cargo test --test cache -- --exact cache_no_reader_block
Testing tcp.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.04s
Running tests/tcp.rs (target/debug/deps/tcp-343222ce4ee4b2ec)
running 1 test
test cancellable_listener_cancel ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Testing thread_pool.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.04s
Running tests/thread_pool.rs (target/debug/deps/thread_pool-4f1bb7d517938467)
running 4 tests
test thread_pool_parallel ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_join_block ... ok
test thread_pool_drop_block ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.06s
Running with cargo --release...
Testing basic functionalities of cache.rs...
Finished `release` profile [optimized] target(s) in 0.03s
Running tests/cache.rs (target/release/deps/cache-5610340c0b6d1bed)
running 1 test
test cache_no_duplicate_sequential ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
Finished `release` profile [optimized] target(s) in 0.03s
Running tests/cache.rs (target/release/deps/cache-5610340c0b6d1bed)
running 1 test
test cache_no_duplicate_concurrent ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.01s
Testing tcp.rs...
Finished `release` profile [optimized] target(s) in 0.03s
Running tests/tcp.rs (target/release/deps/tcp-3f622f7dd3aec98a)
running 1 test
test cancellable_listener_cancel ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Testing thread_pool.rs...
Finished `release` profile [optimized] target(s) in 0.03s
Running tests/thread_pool.rs (target/release/deps/thread_pool-add72be625ad535a)
running 4 tests
test thread_pool_parallel ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_join_block ... ok
test thread_pool_drop_block ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.05s
Running with cargo_asan...
Testing basic functionalities of cache.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8)
running 1 test
test cache_no_duplicate_sequential ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8)
running 1 test
test cache_no_duplicate_concurrent ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.11s
Testing tcp.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
Running tests/tcp.rs (target/x86_64-unknown-linux-gnu/debug/deps/tcp-3b819169d687e46a)
running 1 test
test cancellable_listener_cancel ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
Testing thread_pool.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
Running tests/thread_pool.rs (target/x86_64-unknown-linux-gnu/debug/deps/thread_pool-811f0a12a0be79b6)
running 4 tests
test thread_pool_parallel ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_drop_block ... ok
test thread_pool_join_block ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.07s
Running with cargo_asan --release...
Testing basic functionalities of cache.rs...
Finished `release` profile [optimized] target(s) in 0.07s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/release/deps/cache-e176c6053387d7e7)
running 1 test
test cache_no_duplicate_sequential ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
Finished `release` profile [optimized] target(s) in 0.08s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/release/deps/cache-e176c6053387d7e7)
running 1 test
test cache_no_duplicate_concurrent ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.03s
Testing tcp.rs...
Finished `release` profile [optimized] target(s) in 0.08s
Running tests/tcp.rs (target/x86_64-unknown-linux-gnu/release/deps/tcp-baa0928a468c4699)
running 1 test
test cancellable_listener_cancel ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Testing thread_pool.rs...
Finished `release` profile [optimized] target(s) in 0.07s
Running tests/thread_pool.rs (target/x86_64-unknown-linux-gnu/release/deps/thread_pool-21e837687b5387d2)
running 4 tests
test thread_pool_parallel ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_drop_block ... ok
test thread_pool_join_block ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.06s
Running with cargo_tsan...
Testing basic functionalities of cache.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8)
running 1 test
test cache_no_duplicate_sequential ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8)
running 1 test
test cache_no_duplicate_concurrent ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.11s
Testing tcp.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
Running tests/tcp.rs (target/x86_64-unknown-linux-gnu/debug/deps/tcp-3b819169d687e46a)
running 1 test
test cancellable_listener_cancel ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Testing thread_pool.rs...
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
Running tests/thread_pool.rs (target/x86_64-unknown-linux-gnu/debug/deps/thread_pool-811f0a12a0be79b6)
running 4 tests
test thread_pool_drop_block ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_join_block ... ok
test thread_pool_parallel ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.14s
Running with cargo_tsan --release...
Testing basic functionalities of cache.rs...
Finished `release` profile [optimized] target(s) in 0.07s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/release/deps/cache-e176c6053387d7e7)
running 1 test
test cache_no_duplicate_sequential ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
Finished `release` profile [optimized] target(s) in 0.06s
Running tests/cache.rs (target/x86_64-unknown-linux-gnu/release/deps/cache-e176c6053387d7e7)
running 1 test
test cache_no_duplicate_concurrent ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.04s
Testing tcp.rs...
Finished `release` profile [optimized] target(s) in 0.06s
Running tests/tcp.rs (target/x86_64-unknown-linux-gnu/release/deps/tcp-baa0928a468c4699)
running 1 test
test cancellable_listener_cancel ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Testing thread_pool.rs...
Finished `release` profile [optimized] target(s) in 0.07s
Running tests/thread_pool.rs (target/x86_64-unknown-linux-gnu/release/deps/thread_pool-21e837687b5387d2)
running 4 tests
test thread_pool_drop_block ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_join_block ... ok
test thread_pool_parallel ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.12s
Score: 75 / 100```
GWS0428 commented
And... I ran the test again and achieved a full score this time. I modified one line of code, which might have affected the test result. However, I don't understand why I received a partial score before, even though all tests had passed. It's strange... :(
Lee-Janggun commented
If you look at the log, you have a 'test timed out', which is the failed test. If a test fails without sanitizers, we don't run it with sanitizers. This is why you didn't see the tests being run near the end of the log.
GWS0428 commented
Oh I see..! Thank you very much for your help! Fortunately, my modification achieved a full score without receiving a timeout message.
Additionally, I am curious to know: If I achieve a full score on gg, does that mean I got 100/100? Or will there be another evaluation (running test again, or extra test cases) of my code after the due date?
Lee-Janggun commented
If you got a "Yes" on the submission status and the score says 100/100 on the log, you got a 100/100