Compilation error on Debian 11
airween opened this issue · 8 comments
Hi,
I'm trying to build waflz on a Debian 11 system. During the flow I got this error message:
[ 16%] Building C object src/CMakeFiles/is2.dir/__/ext/udns-0.4/udns_rr_ptr.c.o
/home/airween/src/waflz/src/modsecurity/config_parser.cc:20:10: fatal error: rule.pb.h: No such file or directory
20 | #include "rule.pb.h"
| ^~~~~~~~~~~
compilation terminated.
make[2]: *** [src/CMakeFiles/waflz_obj.dir/build.make:459: src/CMakeFiles/waflz_obj.dir/modsecurity/config_parser.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
The system is up-to-date, all written requirements are satisfied.
What do I miss?
Installed protobuf-compiler package and run this command:
protoc -I=./ --cpp_out=./ ./proto/rule.proto
solved the problem above.
Had to change method in scopes.cc:
diff --git a/src/core/scopes.cc b/src/core/scopes.cc
index 2c263e4..9f021de 100644
--- a/src/core/scopes.cc
+++ b/src/core/scopes.cc
@@ -1111,7 +1111,7 @@ int32_t scopes::process_request_plugin(void **ao_enf, size_t *ao_enf_len,
#if defined(__APPLE__) || defined(__darwin__)
size_t l_enf_size = l_enf->ByteSizeLong();
#else
- size_t l_enf_size = l_enf->ByteSize();
+ size_t l_enf_size = l_enf->ByteSizeLong();
#endif
void *l_enf_buffer = malloc(l_enf_size);
l_enf->SerializeToArray(l_enf_buffer, l_enf_size);
@@ -1123,7 +1123,7 @@ int32_t scopes::process_request_plugin(void **ao_enf, size_t *ao_enf_len,
#if defined(__APPLE__) || defined(__darwin__)
size_t l_a_e_size = l_audit_event->ByteSizeLong();
#else
- size_t l_a_e_size = l_audit_event->ByteSize();
+ size_t l_a_e_size = l_audit_event->ByteSizeLong();
#endif
void *l_a_e_buffer = malloc(l_a_e_size);
l_audit_event->SerializeToArray(l_a_e_buffer, l_a_e_size);
@@ -1135,7 +1135,7 @@ int32_t scopes::process_request_plugin(void **ao_enf, size_t *ao_enf_len,
#if defined(__APPLE__) || defined(__darwin__)
size_t l_p_e_size = l_prod_event->ByteSizeLong();
#else
- size_t l_p_e_size = l_prod_event->ByteSize();
+ size_t l_p_e_size = l_prod_event->ByteSizeLong();
#endif
void *l_p_e_buffer = malloc(l_p_e_size);
l_prod_event->SerializeToArray(l_p_e_buffer, l_p_e_size);and compilation finished. Now I get a test error:
70% tests passed, 12 tests failed out of 40
Total Test time (real) = 12.90 sec
The following tests FAILED:
29 - blackbox_acl (Failed)
30 - blackbox_acl_accesslists (Failed)
31 - blackbox_ruleset (Failed)
32 - blackbox_rules (Failed)
33 - blackbox_rtu (Failed)
34 - blackbox_profile (Failed)
35 - blackbox_wjc (Failed)
36 - blackbox_challenge (Failed)
37 - blackbox_scopes (Failed)
38 - blackbox_scopes_updates (Failed)
39 - blackbox_limit_lmdb (Failed)
40 - blackbox_limit (Failed)
Errors while running CTest
make: *** [Makefile:171: test] Error 8
Any ideas about the reason?
The black-box tests require python and a few modules.
I think the current module requirements are in the requirements.txt file.
With regard to the ByteSizeLong(... changes. I'll create a branch now to see if it'll pass our CI/CD.
Thanks so much for bringing this to our attention!
Too bad this didn't turn out to be as trivial as I'd hoped -see:
For 18.04 getting this error now:
[ 42%] Building CXX object src/CMakeFiles/waflz_obj.dir/core/scopes.cc.o
/home/runner/work/waflz/waflz/src/core/scopes.cc: In member function ‘int32_t ns_waflz::scopes::process_request_plugin(void**, size_t*, void**, size_t*, void**, size_t*, void*, const ns_waflz::rqst_ctx_callbacks*, ns_waflz::rqst_ctx**)’:
/home/runner/work/waflz/waflz/src/core/scopes.cc:1121:44: error: ‘const class waflz_pb::enforcement’ has no member named ‘ByteSizeLong’; did you mean ‘ByteSize’?
size_t l_enf_size = l_enf->ByteSizeLong();
^~~~~~~~~~~~
ByteSize
...I'll need to figure out how to check for a version. I'll keep plugging away at this.
@airween if you get a sec can you git pull master and retry the build? Hopefully builds without modifications now on Debian 11.
Thanks so much.
thanks for all feedback!
@airween if you get a sec can you
git pullmaster and retry the build? Hopefully builds without modifications now on Debian 11.
I did it, the code successfully compiled without any modifications.
It isn't mentioned, but looks like build.sh uses Python2. The required packages were satisfied for Python3 only:
$ pip install -r requirements.txt
Requirement already satisfied: pytest>=3.0.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 1)) (6.0.2)
Requirement already satisfied: pytest-sugar>=0.8.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (0.9.4)
Requirement already satisfied: requests>=2.20.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 3)) (2.25.1)so I upgraded all Python2 related packages.
Now I get:
30/40 Test #30: blackbox_acl_accesslists ......... Passed 2.06 sec
Start 31: blackbox_ruleset
31/40 Test #31: blackbox_ruleset .................***Failed 2.20 sec
Start 32: blackbox_rules
32/40 Test #32: blackbox_rules ................... Passed 2.28 sec
Start 33: blackbox_rtu
...
...
98% tests passed, 1 tests failed out of 40
Total Test time (real) = 135.49 sec
The following tests FAILED:
31 - blackbox_ruleset (Failed)
Errors while running CTest
make: *** [Makefile:171: test] Error 8
Thanks so much.
I thank you :).
Also a note, that I checked the build process on Ubuntu 20.04, but there I also get an error:
Scanning dependencies of target subrequest_cmd
[ 96%] Building CXX object examples/CMakeFiles/subrequest_cmd.dir/subrequest.cc.o
[ 96%] Built target proxy_cmd
Scanning dependencies of target w_stat_cmd
[ 97%] Building CXX object examples/CMakeFiles/w_stat_cmd.dir/w_stat.cc.o
[ 98%] Linking CXX executable subrequest
[100%] Linking CXX executable w_stat
[100%] Built target subrequest_cmd
[100%] Built target w_stat_cmd
[ 5%] No install step for 'submodule_is2'
[ 5%] Completed 'submodule_is2'
[ 5%] Built target submodule_is2
make: *** [Makefile:163: all] Error 2
Just FYI.
For the compile error on 20.04 can you show more of the output? It's curious because our CI/CD for github actions requires all pull-requests to pass building and testing for 18.04/20.04/OS X(latest).
See:
https://github.com/EdgeCast/waflz/blob/master/.github/workflows/build.yml
As for the test suite running python2, I think it's just calling pytest, so it should be whatever pytest is being used.
See:
https://github.com/EdgeCast/waflz/blob/master/tests/blackbox/run_tests.sh#L13
We've tried to uninstall python2 from all of our production and developer environments, so maybe we hadn't noticed it. Maybe we need to explicitly require it or run pip3 install -r requirements ?
As for the particular test case failing, you can run an individual test from the test suit in the test dir like this and maybe get more info/output.
~/gproj/waflz/tests/blackbox>pytest -v ./ruleset/test_ruleset.py
Test session starts (platform: linux, Python 3.8.10, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /home/rmorrison/gproj/waflz/tests/blackbox, configfile: pytest.ini
plugins: sugar-0.9.4
collecting ...
ruleset/test_ruleset.py::test_OWASP_2_2_9_anomaly ✓ 25% ██▌
ruleset/test_ruleset.py::test_OWASP_2_2_9_anomaly_low ✓ 50% █████
ruleset/test_ruleset.py::test_OWASP_3_2_anomaly ✓ 75% ███████▌
ruleset/test_ruleset.py::test_OWASP_3_2_anomaly_low ✓ 100% ██████████
Results (1.83s):
4 passedso just pytest <some_test.py> in the test dir.
Thanks so much for all the insights!
For the compile error on 20.04 can you show more of the output? It's curious because our CI/CD for github actions requires all pull-requests to pass building and testing for 18.04/20.04/OS X(latest). See: https://github.com/EdgeCast/waflz/blob/master/.github/workflows/build.yml
thanks, that helped me. Otherwise, I reviewed the build process on Ubuntu 20.04, and realized that there was a message:
CMake Warning at /usr/share/cmake-3.16/Modules/FindProtobuf.cmake:499 (message):
Protobuf compiler version doesn't match library version 3.6.1
I looked up this message, and found that I have to install some additional packages. Summary, this is what I made on a (nearly) pure Ubuntu 20.04 machine:
$ sudo apt install libprotobuf-dev protobuf-compiler python-is-python3 python3-pip
$ sudo pip3 install -r requirements.txt
For some reason, the pip3 install without sudo wasn't enough.
Now I get:
...
100% tests passed, 0 tests failed out of 40
...
Run CPack packaging tool...
CPack: Create package using DEB
CPack: Install projects
CPack: - Run preinstall target for: waflz
CPack: - Install project: waflz []
CPack: Create package
-- CPACK_DEBIAN_PACKAGE_DEPENDS not set, the package will have no dependencies.
CPack: - package: /home/airween/src/waflz/build/waflz_1.12.2-1-gd64b28e-focal_amd64.deb generated.
Scanning dependencies of target release
: extracting deb: /home/airween/src/waflz/build/waflz_1.12.2-1-gd64b28e-focal_amd64.deb
: extract to: /tmp/deb2tar_24hz1Wme
: create bundle dir: /tmp/deb2tar_jAeFm2Bd
: make os dir: /tmp/deb2tar_jAeFm2Bd/ubuntu20.04-x86_64
: creating tarball...
: created: /home/airween/src/waflz/build/waflz_1.12.2-1-gd64b28e-focal_amd64.tar.gz
: cleaning up
Built target release
~/src/waflz
So, looks like everything is fine.
As for the test suite running python2, I think it's just calling
pytest, so it should be whateverpytestis being used. See: https://github.com/EdgeCast/waflz/blob/master/tests/blackbox/run_tests.sh#L13
may be that's a Debian issue, but I could solve that too.
We've tried to uninstall python2 from all of our production and developer environments, so maybe we hadn't noticed it. Maybe we need to explicitly require it or run
pip3 install -r requirements?
On Debian, I had to remove the python3-pytest package, and had to install it through pip3; also had to remove the Python2's pytest.
$ sudo dpkg -r python3-pytest python3-pytest-sugar
$ pip uninstall pytest
$ sudo apt install python-is-python3
$ pip3 install -r requirements.txt
After this, everything is fine here too.
so just
pytest <some_test.py>in the test dir.
Thanks.
Thanks so much for all the insights!
Thank you again, now I was able to build WAFlz package - and I can start to play with it :).
I'm closing this issue - thanks for all of your help.