Running the tests failed
liangcheng-yu opened this issue · 2 comments
liangcheng-yu commented
Hi, I was trying to set up ubpf on a clean Ubuntu 20.04.6 LTS, everything went well but the tests failed. Below are the details:
sudo apt-get update
sudo apt install cmake # version == 3.16.3
sudo apt install python2
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
sudo python2 get-pip.py
git clone https://github.com/iovisor/ubpf/
git submodule init
git submodule update --recursive
pip2 install -r requirements.txt
sudo apt-get install libbpf-dev
sudo apt-get install libboost-all-dev
git clone https://github.com/serge1/ELFIO.git
sudo mv ELFIO/elfio/ /usr/local/include/
sudo apt-get install clang # clang version 10.0.0, newer than 3.7
cmake -S . -B build -DUBPF_ENABLE_TESTS=true
cmake --build build --config Debug
cmake --build build --target test -- # This is the step that gets wrongI got the following error:
Running tests...
Test project /home/ubuntu/ubpf/build
Start 1: cpu_v1
1/38 Test #1: cpu_v1 ...........................***Failed 0.10 sec
Start 2: cpu_v2
2/38 Test #2: cpu_v2 ...........................***Failed 0.01 sec
Start 3: cpu_v3
3/38 Test #3: cpu_v3 ...........................***Failed 0.00 sec
Start 4: elf_format
4/38 Test #4: elf_format .......................***Failed 0.00 sec
Start 5: no_bpf_bytecode
5/38 Test #5: no_bpf_bytecode ..................***Failed Required regular expression not found. Regex=[Test file has no BPF instructions
] 0.00 sec
Start 6: invalid_return_value
6/38 Test #6: invalid_return_value .............***Failed Required regular expression not found. Regex=[Plugin returned invalid return value Hello World
] 0.00 sec
Start 7: wrong_output
7/38 Test #7: wrong_output .....................***Failed Required regular expression not found. Regex=[Plugin returned incorrect return value 1 expected 3
] 0.00 sec
Start 8: invalid_plugin_name
8/38 Test #8: invalid_plugin_name ..............***Failed Required regular expression not found. Regex=[failed to execute test
] 0.00 sec
Start 9: plugin_failed
9/38 Test #9: plugin_failed ....................***Failed Required regular expression not found. Regex=[Plugin returned error code 2
] 0.00 sec
Start 10: wrong_error
10/38 Test #10: wrong_error ......................***Failed Required regular expression not found. Regex=[Plugin returned error code 1 and output but expected Invalid
] 0.00 sec
Start 11: expect_failure
11/38 Test #11: expect_failure ...................***Failed Required regular expression not found. Regex=[Plugin returned invalid return value This is a negative test
] 0.00 sec
Start 12: verifier_failure
12/38 Test #12: verifier_failure .................***Failed Required regular expression not found. Regex=[Plugin returned error code 1 and output Failed to load program
] 0.00 sec
Start 13: invalid_register
13/38 Test #13: invalid_register .................***Failed Required regular expression not found. Regex=[Invalid register: %r50
] 0.00 sec
Start 14: invalid_offset
14/38 Test #14: invalid_offset ...................***Failed Required regular expression not found. Regex=[Failed to decode register and offset: %r1
] 0.01 sec
Start 15: invalid_operand_count
15/38 Test #15: invalid_operand_count ............***Failed Required regular expression not found. Regex=[Invalid number of operands for mnemonic: lddw
] 0.00 sec
Start 16: invalid_label
16/38 Test #16: invalid_label ....................***Failed Required regular expression not found. Regex=[Invalid label: NOT_A_LABEL
] 0.00 sec
Start 17: invalid_mnemonic
17/38 Test #17: invalid_mnemonic .................***Failed Required regular expression not found. Regex=[Invalid mnemonic: ldxq
] 0.01 sec
Start 18: libbpf_plugin_usage
18/38 Test #18: libbpf_plugin_usage ..............***Failed Required regular expression not found. Regex=[usage:
] 0.01 sec
Start 19: invalid_lock
19/38 Test #19: invalid_lock .....................***Failed Required regular expression not found. Regex=[Invalid number of operands
] 0.01 sec
Start 20: invalid_lock2
20/38 Test #20: invalid_lock2 ....................***Failed Required regular expression not found. Regex=[Invalid number of operands for lock
] 0.01 sec
Start 21: include_test
21/38 Test #21: include_test .....................***Failed Required regular expression not found. Regex=[PASS
] 0.01 sec
Start 22: exclude_test
22/38 Test #22: exclude_test .....................***Failed Required regular expression not found. Regex=[SKIP
] 0.00 sec
Start 23: negative_include_test
23/38 Test #23: negative_include_test ............***Failed Required regular expression not found. Regex=[SKIP
] 0.01 sec
Start 24: negative_exclude_test
24/38 Test #24: negative_exclude_test ............***Failed 0.00 sec
Start 25: usage
25/38 Test #25: usage ............................***Failed Required regular expression not found. Regex=[Options:
] 0.01 sec
Start 26: missing_test_path
26/38 Test #26: missing_test_path ................***Failed Required regular expression not found. Regex=[test_file_path or test_file_directory
] 0.01 sec
Start 27: missing_plugin_path
27/38 Test #27: missing_plugin_path ..............***Failed Required regular expression not found. Regex=[plugin_path is required
] 0.01 sec
Start 28: invalid_cpu_version
28/38 Test #28: invalid_cpu_version ..............***Failed Required regular expression not found. Regex=[Invalid CPU version
] 0.00 sec
Start 29: invalid_test_directory
29/38 Test #29: invalid_test_directory ...........***Failed Required regular expression not found. Regex=[No such file or directory
] 0.00 sec
Start 30: fail_with_error
30/38 Test #30: fail_with_error ..................***Failed Required regular expression not found. Regex=[Plugin returned error code 1 and output
] 0.01 sec
Start 31: unknown_directive
31/38 Test #31: unknown_directive ................***Failed Required regular expression not found. Regex=[Test file has no BPF instructions.
] 0.01 sec
Start 32: libbpf_program_bytes
32/38 Test #32: libbpf_program_bytes .............***Failed Required regular expression not found. Regex=[0
] 0.01 sec
Start 33: invalid_option
Could not find executable /home/ubuntu/ubpf/build/external/bpf_conformance/bin/bpf_conformance_runner
Looked in the following places:
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/Release/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/Release/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/Debug/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/Debug/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/MinSizeRel/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/MinSizeRel/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/RelWithDebInfo/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/RelWithDebInfo/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/Deployment/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/Deployment/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/Development/bpf_conformance_runner
/home/ubuntu/ubpf/build/external/bpf_conformance/bin/Development/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/Release/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/Release/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/Debug/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/Debug/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/MinSizeRel/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/MinSizeRel/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/RelWithDebInfo/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/RelWithDebInfo/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/Deployment/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/Deployment/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/Development/bpf_conformance_runner
home/ubuntu/ubpf/build/external/bpf_conformance/bin/Development/bpf_conformance_runner
Unable to find executable: /home/ubuntu/ubpf/build/external/bpf_conformance/bin/bpf_conformance_runner
33/38 Test #33: invalid_option ...................***Not Run 0.00 sec
Start 34: list_used_instructions
34/38 Test #34: list_used_instructions ...........***Failed Required regular expression not found. Regex=[Instructions used by tests
] 0.01 sec
Start 35: list_unused_instructions
35/38 Test #35: list_unused_instructions .........***Failed Required regular expression not found. Regex=[Instructions not used by tests
] 0.00 sec
Start 36: back_compat
36/38 Test #36: back_compat ......................***Failed 0.01 sec
Start 37: map_TEST_INTERPRET
37/38 Test #37: map_TEST_INTERPRET ............... Passed 0.00 sec
Start 38: map_TEST_JIT
38/38 Test #38: map_TEST_JIT ..................... Passed 0.00 sec
5% tests passed, 36 tests failed out of 38
Total Test time (real) = 0.32 sec
The following tests FAILED:
1 - cpu_v1 (Failed)
2 - cpu_v2 (Failed)
3 - cpu_v3 (Failed)
4 - elf_format (Failed)
5 - no_bpf_bytecode (Failed)
6 - invalid_return_value (Failed)
7 - wrong_output (Failed)
8 - invalid_plugin_name (Failed)
9 - plugin_failed (Failed)
10 - wrong_error (Failed)
11 - expect_failure (Failed)
12 - verifier_failure (Failed)
13 - invalid_register (Failed)
14 - invalid_offset (Failed)
15 - invalid_operand_count (Failed)
16 - invalid_label (Failed)
17 - invalid_mnemonic (Failed)
18 - libbpf_plugin_usage (Failed)
19 - invalid_lock (Failed)
20 - invalid_lock2 (Failed)
21 - include_test (Failed)
22 - exclude_test (Failed)
23 - negative_include_test (Failed)
24 - negative_exclude_test (Failed)
25 - usage (Failed)
26 - missing_test_path (Failed)
27 - missing_plugin_path (Failed)
28 - invalid_cpu_version (Failed)
29 - invalid_test_directory (Failed)
30 - fail_with_error (Failed)
31 - unknown_directive (Failed)
32 - libbpf_program_bytes (Failed)
33 - invalid_option (Not Run)
34 - list_used_instructions (Failed)
35 - list_unused_instructions (Failed)
36 - back_compat (Failed)
Errors while running CTest
make: *** [Makefile:73: test] Error 8
Noticing that somehow by default the binary for bpf_conformance_runner is at the wrong place, so I manually cp build/bin/bpf_conformance_runner build/external/bpf_conformance/bin/ and got
Running tests...
Test project /home/ubuntu/ubpf/build
Start 1: cpu_v1
1/38 Test #1: cpu_v1 ...........................***Failed 0.32 sec
Start 2: cpu_v2
2/38 Test #2: cpu_v2 ...........................***Failed 0.12 sec
Start 3: cpu_v3
3/38 Test #3: cpu_v3 ...........................***Failed 0.15 sec
Start 4: elf_format
4/38 Test #4: elf_format .......................***Failed 0.16 sec
Start 5: no_bpf_bytecode
5/38 Test #5: no_bpf_bytecode .................. Passed 0.01 sec
Start 6: invalid_return_value
6/38 Test #6: invalid_return_value ............. Passed 0.06 sec
Start 7: wrong_output
7/38 Test #7: wrong_output ..................... Passed 0.01 sec
Start 8: invalid_plugin_name
8/38 Test #8: invalid_plugin_name .............. Passed 0.01 sec
Start 9: plugin_failed
9/38 Test #9: plugin_failed .................... Passed 0.02 sec
Start 10: wrong_error
10/38 Test #10: wrong_error ......................***Failed Required regular expression not found. Regex=[Plugin returned error code 1 and output but expected Invalid
] 0.01 sec
Start 11: expect_failure
11/38 Test #11: expect_failure ................... Passed 0.01 sec
Start 12: verifier_failure
12/38 Test #12: verifier_failure .................***Failed Required regular expression not found. Regex=[Plugin returned error code 1 and output Failed to load program
] 0.01 sec
Start 13: invalid_register
13/38 Test #13: invalid_register ................. Passed 0.01 sec
Start 14: invalid_offset
14/38 Test #14: invalid_offset ................... Passed 0.01 sec
Start 15: invalid_operand_count
15/38 Test #15: invalid_operand_count ............ Passed 0.02 sec
Start 16: invalid_label
16/38 Test #16: invalid_label .................... Passed 0.01 sec
Start 17: invalid_mnemonic
17/38 Test #17: invalid_mnemonic ................. Passed 0.01 sec
Start 18: libbpf_plugin_usage
18/38 Test #18: libbpf_plugin_usage ..............***Failed Required regular expression not found. Regex=[usage:
] 0.01 sec
Start 19: invalid_lock
19/38 Test #19: invalid_lock ..................... Passed 0.01 sec
Start 20: invalid_lock2
20/38 Test #20: invalid_lock2 .................... Passed 0.01 sec
Start 21: include_test
21/38 Test #21: include_test .....................***Failed Required regular expression not found. Regex=[PASS
] 0.01 sec
Start 22: exclude_test
22/38 Test #22: exclude_test ..................... Passed 0.01 sec
Start 23: negative_include_test
23/38 Test #23: negative_include_test ............ Passed 0.01 sec
Start 24: negative_exclude_test
24/38 Test #24: negative_exclude_test ............***Failed 0.01 sec
Start 25: usage
25/38 Test #25: usage ............................ Passed 0.01 sec
Start 26: missing_test_path
26/38 Test #26: missing_test_path ................ Passed 0.01 sec
Start 27: missing_plugin_path
27/38 Test #27: missing_plugin_path .............. Passed 0.01 sec
Start 28: invalid_cpu_version
28/38 Test #28: invalid_cpu_version .............. Passed 0.01 sec
Start 29: invalid_test_directory
29/38 Test #29: invalid_test_directory ........... Passed 0.01 sec
Start 30: fail_with_error
30/38 Test #30: fail_with_error ..................***Failed Required regular expression not found. Regex=[Plugin returned error code 1 and output
] 0.01 sec
Start 31: unknown_directive
31/38 Test #31: unknown_directive ................ Passed 0.01 sec
Start 32: libbpf_program_bytes
32/38 Test #32: libbpf_program_bytes .............***Failed Required regular expression not found. Regex=[0
] 0.01 sec
Start 33: invalid_option
33/38 Test #33: invalid_option ................... Passed 0.00 sec
Start 34: list_used_instructions
34/38 Test #34: list_used_instructions ........... Passed 0.12 sec
Start 35: list_unused_instructions
35/38 Test #35: list_unused_instructions ......... Passed 0.12 sec
Start 36: back_compat
36/38 Test #36: back_compat ......................***Failed 0.01 sec
Start 37: map_TEST_INTERPRET
37/38 Test #37: map_TEST_INTERPRET ............... Passed 0.00 sec
Start 38: map_TEST_JIT
38/38 Test #38: map_TEST_JIT ..................... Passed 0.00 sec
68% tests passed, 12 tests failed out of 38
Total Test time (real) = 1.41 sec
The following tests FAILED:
1 - cpu_v1 (Failed)
2 - cpu_v2 (Failed)
3 - cpu_v3 (Failed)
4 - elf_format (Failed)
10 - wrong_error (Failed)
12 - verifier_failure (Failed)
18 - libbpf_plugin_usage (Failed)
21 - include_test (Failed)
24 - negative_exclude_test (Failed)
30 - fail_with_error (Failed)
32 - libbpf_program_bytes (Failed)
36 - back_compat (Failed)
Errors while running CTest
make: *** [Makefile:73: test] Error 8
It is better than the prior success rate (5%), still some tests failed.
I am wondering about what is missing, e.g., is it necessary to roll back the clang version to 3.7? Thanks!
liangcheng-yu commented
Thanks Alan! The issue is resolved.
liangcheng-yu commented
Hi Alan, I moved to another clean Ubuntu 20.04.06 LTS VM, but somehow there is a few tests failed, as attached below.
Note that I was running the same set up as before (which gave me 100% tests passed), it seems to be caused by some minor reason between the two VMs.
611/630 Test #611: invalid_lock ....................................................................... Passed 0.01 sec
Start 612: invalid_lock2
612/630 Test #612: invalid_lock2 ...................................................................... Passed 0.01 sec
Start 613: include_test
613/630 Test #613: include_test .......................................................................***Failed Required regular expression not found. Regex=[PASS
] 0.01 sec
Start 614: exclude_test
614/630 Test #614: exclude_test ....................................................................... Passed 0.02 sec
Start 615: negative_include_test
615/630 Test #615: negative_include_test .............................................................. Passed 0.01 sec
Start 616: negative_exclude_test
616/630 Test #616: negative_exclude_test ..............................................................***Failed 0.01 sec
Start 617: usage
617/630 Test #617: usage .............................................................................. Passed 0.01 sec
Start 618: missing_test_path
618/630 Test #618: missing_test_path .................................................................. Passed 0.01 sec
Start 619: missing_plugin_path
619/630 Test #619: missing_plugin_path ................................................................ Passed 0.01 sec
Start 620: invalid_cpu_version
620/630 Test #620: invalid_cpu_version ................................................................ Passed 0.01 sec
Start 621: invalid_test_directory
621/630 Test #621: invalid_test_directory ............................................................. Passed 0.01 sec
Start 622: fail_with_error
622/630 Test #622: fail_with_error .................................................................... Passed 0.01 sec
Start 623: unknown_directive
623/630 Test #623: unknown_directive .................................................................. Passed 0.01 sec
Start 624: libbpf_program_bytes
624/630 Test #624: libbpf_program_bytes ...............................................................***Failed Required regular expression not found. Regex=[0
] 0.01 sec
Start 625: invalid_option
625/630 Test #625: invalid_option ..................................................................... Passed 0.00 sec
Start 626: list_used_instructions
626/630 Test #626: list_used_instructions ............................................................. Passed 0.29 sec
Start 627: list_unused_instructions
627/630 Test #627: list_unused_instructions ........................................................... Passed 0.28 sec
Start 628: back_compat
628/630 Test #628: back_compat ........................................................................***Failed 0.01 sec
Start 629: map_TEST_INTERPRET
629/630 Test #629: map_TEST_INTERPRET ................................................................. Passed 0.00 sec
Start 630: map_TEST_JIT
630/630 Test #630: map_TEST_JIT ....................................................................... Passed 0.00 sec
99% tests passed, 8 tests failed out of 630
Total Test time (real) = 7.54 sec
The following tests FAILED:
593 - cpu_v1 (Failed)
594 - cpu_v2 (Failed)
595 - cpu_v3 (Failed)
596 - elf_format (Failed)
613 - include_test (Failed)
616 - negative_exclude_test (Failed)
624 - libbpf_program_bytes (Failed)
628 - back_compat (Failed)
Errors while running CTest
make: *** [Makefile:73: test] Error 8
What's odd is that the two VMs have the same exact OS release (20.04.6 LTS (Focal Fossa)), clang version etc.
The only minor difference is that the VM with 100% tests passed has a slightly different kernel verision:
# The one with 100% tests
ubuntu@ip-172-31-80-18:~/ubpf$ uname -a
Linux ip-172-31-80-18 5.15.0-1036-aws #40~20.04.1-Ubuntu SMP Mon Apr 24 00:21:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
# The one with 99% tests
lc@vm1:~/ubpf/programs$ uname -a
Linux rcf-vm1 5.15.0-1038-azure #45~20.04.1-Ubuntu SMP Tue Apr 25 18:45:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux