Python.h file not found
stormshawn opened this issue · 14 comments
stormshawn commented
I keep getting the error:
In file included from /home/mizkulg/anaconda3/envs/pybind/lib/python3.7/site-packages/pybind11/include/pybind11/buffer_info.h:12,
from /usr/local/include/carma/carma/converters.h:22,
from /usr/local/include/carma/carma/arraystore.h:1,
from /usr/local/include/carma/carma.h:1,
from carma.cpp:3:
/home/mizkukg/anaconda3/envs/pybind/lib/python3.7/site-packages/pybind11/include/pybind11/detail/common.h:112:10: fatal error: Python.h: No such file or directory
#include <Python.h>
^~~~~~~~~~
compilation terminated.
Conda 4.8.2
Python 3.7.3
Pybind11 runs fine on other programs that I have. But with carma the example code at examples/example.cpp gives me the error above. I am running:
c++ -O3 -Wall -shared -std=c++14 -fPIC -larmadillo python3 -m pybind11 --includes carma.cpp -o example python3-config --extension-suffix
I've tried all the recommendations on stackoverflow but nothing that I install resolves this problem. Any insight is appreciated.
RUrlus commented
OK, surprising that you don't have this issue with other pybind11 builds. Are you on the same version of Pybind for those as well?
My initial thought would be that you're calling the Conda python and not the python3-dev through apt.
We don't do anything with Python.h so I am not entirely sure how this could be due to something we are doing.
Have you tried building the examples with CMake?
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_EXAMPLES=true .. && make install
stormshawn commented
Yea same conda environment and nearly same commands as the one above (c++ -03...). I didn't find anything on the conda being a problem with python3-dev. Since other pybind11 builds, I'm not sure if that would be the problem.
I just tried the CMake and I get the following error:
CMake Error at CMakeLists.txt:109 (add_subdirectory):
The source directory
/home/mizkulg/carma/third_party/pybind11
does not contain a CMakeLists.txt file.
CMake Error at examples/CMakeLists.txt:8 (pybind11_add_module):
Unknown CMake command "pybind11_add_module".
-- Configuring incomplete, errors occurred!
See also "/home/mizkulg/carma/build/CMakeFiles/CMakeOutput.log".
RUrlus commented
Can you run git submodule update --init in the Carma repo? You're missing the submodules
stormshawn commented
Ok just did and ran the above command again and am getting:
CMake Error: The source directory "/home/shawn" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
RUrlus commented
Go to the root of the Carma repo and execute line by line:
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_EXAMPLES=true -DBUILD_TESTS=true .. && make install
ctest
stormshawn commented
Here are the results:
~/carma/build master ❯ cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_EXAMPLES=true -DBUILD_TESTS=true .. && make install
-- Found PythonInterp: /home/mizkulg/anaconda3/envs/pybind/bin/python3.7 (found version "3.7.3")
-- Found PythonLibs: /home/mizkulg/anaconda3/envs/pybind/lib/libpython3.7m.so
-- pybind11 v2.4.3
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- clang-format target for updating code format is available
-- Configuring done
-- Generating done
-- Build files have been written to: /home/mizkulg/carma/build
Scanning dependencies of target test_carma
[ 7%] Building CXX object tests/CMakeFiles/test_carma.dir/src/bindings.cpp.o
[ 15%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_main.cpp.o
[ 23%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_store.cpp.o
[ 30%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_nparray.cpp.o
[ 38%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_arr_to_mat.cpp.o
[ 46%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_mat_to_arr.cpp.o
[ 53%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_type_caster.cpp.o
[ 61%] Linking CXX shared module test_carma.cpython-37m-x86_64-linux-gnu.so
[ 61%] Built target test_carma
Scanning dependencies of target example_carma
[ 69%] Building CXX object examples/CMakeFiles/example_carma.dir/arraystore.cpp.o
In file included from /home/mizkulg/carma/third_party/pybind11/include/pybind11/buffer_info.h:12,
from /home/mizkulg/carma/include/carma/carma/converters.h:22,
from /home/mizkulg/carma/include/carma/carma/arraystore.h:1,
from /home/mizkulg/carma/include/carma/carma.h:1,
from /home/mizkulg/carma/examples/arraystore.h:2,
from /home/mizkulg/carma/examples/arraystore.cpp:1:
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:681:83: warning: inline declaration of ‘void pybind11::pybind11_fail(const string&)’ follows declaration with attribute ‘noinline’ [-Wattributes]
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); }
^
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:680:44: note: previous definition of ‘void pybind11::pybind11_fail(const char*)’ was here
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); }
^~~~~~~~~~~~~
[ 76%] Building CXX object examples/CMakeFiles/example_carma.dir/example_bindings.cpp.o
[ 84%] Building CXX object examples/CMakeFiles/example_carma.dir/manual_conversion.cpp.o
In file included from /home/mizkulg/carma/third_party/pybind11/include/pybind11/buffer_info.h:12,
from /home/mizkulg/carma/include/carma/carma/converters.h:22,
from /home/mizkulg/carma/include/carma/carma/arraystore.h:1,
from /home/mizkulg/carma/include/carma/carma.h:1,
from /home/mizkulg/carma/examples/manual_conversion.h:2,
from /home/mizkulg/carma/examples/manual_conversion.cpp:1:
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:681:83: warning: inline declaration of ‘void pybind11::pybind11_fail(const string&)’ follows declaration with attribute ‘noinline’ [-Wattributes]
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); }
^
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:680:44: note: previous definition of ‘void pybind11::pybind11_fail(const char*)’ was here
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); }
^~~~~~~~~~~~~
[ 92%] Building CXX object examples/CMakeFiles/example_carma.dir/automatic_conversion.cpp.o
In file included from /home/mizkulg/carma/third_party/pybind11/include/pybind11/buffer_info.h:12,
from /home/mizkulg/carma/include/carma/carma/converters.h:22,
from /home/mizkulg/carma/include/carma/carma/arraystore.h:1,
from /home/mizkulg/carma/include/carma/carma.h:1,
from /home/mizkulg/carma/examples/automatic_conversion.h:2,
from /home/mizkulg/carma/examples/automatic_conversion.cpp:1:
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:681:83: warning: inline declaration of ‘void pybind11::pybind11_fail(const string&)’ follows declaration with attribute ‘noinline’ [-Wattributes]
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); }
^
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:680:44: note: previous definition of ‘void pybind11::pybind11_fail(const char*)’ was here
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); }
^~~~~~~~~~~~~
[100%] Linking CXX shared module example_carma.cpython-37m-x86_64-linux-gnu.so
[100%] Built target example_carma
Install the project...
-- Install configuration: "Debug"
-- Installing: /home/mizkulg/carma/build/./include
-- Installing: /home/mizkulg/carma/build/./include/carma
-- Installing: /home/mizkulg/carma/build/./include/carma/carma
-- Installing: /home/mizkulg/carma/build/./include/carma/carma/nparray.h
-- Installing: /home/mizkulg/carma/build/./include/carma/carma/converters.h
-- Installing: /home/mizkulg/carma/build/./include/carma/carma/arraystore.h
-- Installing: /home/mizkulg/carma/build/./include/carma/carma/utils.h
-- Installing: /home/mizkulg/carma/build/./include/carma/carma.h
-- Installing: /home/mizkulg/carma/build/tests/test_arr_to_mat.py
-- Installing: /home/mizkulg/carma/build/tests/test_arraystore.py
-- Installing: /home/mizkulg/carma/build/tests/test_loading.py
-- Installing: /home/mizkulg/carma/build/tests/test_mat_to_arr.py
-- Installing: /home/mizkulg/carma/build/tests/test_nparray.py
-- Installing: /home/mizkulg/carma/build/tests/test_type_caster.py
-- Installing: /home/mizkulg/carma/build/examples/carma_examples.py
CMake Error at cmake_install.cmake:61 (file):
file failed to open for writing (Permission denied):
/home/mizkulg/carma/build/install_manifest.txt
make: *** [Makefile:129: install] Error 1
~/carma/build master ❯ ctest 1m 19s Py pybind 15:53:43
Test project /home/mizkulg/carma/build
Start 1: pytest
1/2 Test #1: pytest ...........................***Failed 0.09 sec
Start 2: example
2/2 Test #2: example ..........................***Failed 0.05 sec
0% tests passed, 2 tests failed out of 2
Total Test time (real) = 0.14 sec
The following tests FAILED:
1 - pytest (Failed)
2 - example (Failed)
Errors while running CTest
RUrlus commented
Make Error at cmake_install.cmake:61 (file):
file failed to open for writing (Permission denied):
you don't have the correct permissions for some reason for this directory.
can you run:
rm build
mkdir build
sudo chmod 777 build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_EXAMPLES=true -DBUILD_TESTS=true .. && make install
ctest
RUrlus commented
Regardless of the above, the tests and examples compile. I am not sure what is happening with the manual compilation but I think it's a mismatch between the python version in conda and what is being picked up by pybind11.
Doesn't look like it's an issue with Carma.
BTW you're missing backticks:
c++ -O3 -Wall -shared -std=c++14 -fPIC -larmadillo `python3 -m pybind11 --includes` example.cpp -o example `python3-config --extension-suffix`
stormshawn commented
The results also failed:
~/carma/build master ❯ cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=. -DBUILD_EXAMPLES=true -DBUILD_TESTS=true .. && make install
-- The CXX compiler identification is GNU 8.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /home/mizkulg/anaconda3/envs/pybind/bin/python3.7 (found version "3.7.3")
-- Found PythonLibs: /home/mizkulg/anaconda3/envs/pybind/lib/libpython3.7m.so
-- pybind11 v2.4.3
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- clang-format target for updating code format is available
-- Configuring done
-- Generating done
-- Build files have been written to: /home/mizkulg/carma/build
Scanning dependencies of target test_carma
[ 7%] Building CXX object tests/CMakeFiles/test_carma.dir/src/bindings.cpp.o
[ 15%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_main.cpp.o
[ 23%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_store.cpp.o
[ 30%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_nparray.cpp.o
[ 38%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_arr_to_mat.cpp.o
[ 46%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_mat_to_arr.cpp.o
[ 53%] Building CXX object tests/CMakeFiles/test_carma.dir/src/test_type_caster.cpp.o
[ 61%] Linking CXX shared module test_carma.cpython-37m-x86_64-linux-gnu.so
[ 61%] Built target test_carma
Scanning dependencies of target example_carma
[ 69%] Building CXX object examples/CMakeFiles/example_carma.dir/arraystore.cpp.o
In file included from /home/mizkulg/carma/third_party/pybind11/include/pybind11/buffer_info.h:12,
from /home/mizkulg/carma/include/carma/carma/converters.h:22,
from /home/mizkulg/carma/include/carma/carma/arraystore.h:1,
from /home/mizkulg/carma/include/carma/carma.h:1,
from /home/mizkulg/carma/examples/arraystore.h:2,
from /home/mizkulg/carma/examples/arraystore.cpp:1:
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:681:83: warning: inline declaration of ‘void pybind11::pybind11_fail(const string&)’ follows declaration with attribute ‘noinline’ [-Wattributes]
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); }
^
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:680:44: note: previous definition of ‘void pybind11::pybind11_fail(const char*)’ was here
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); }
^~~~~~~~~~~~~
[ 76%] Building CXX object examples/CMakeFiles/example_carma.dir/example_bindings.cpp.o
[ 84%] Building CXX object examples/CMakeFiles/example_carma.dir/manual_conversion.cpp.o
In file included from /home/mizkulg/carma/third_party/pybind11/include/pybind11/buffer_info.h:12,
from /home/mizkulg/carma/include/carma/carma/converters.h:22,
from /home/mizkulg/carma/include/carma/carma/arraystore.h:1,
from /home/mizkulg/carma/include/carma/carma.h:1,
from /home/mizkulg/carma/examples/manual_conversion.h:2,
from /home/mizkulg/carma/examples/manual_conversion.cpp:1:
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:681:83: warning: inline declaration of ‘void pybind11::pybind11_fail(const string&)’ follows declaration with attribute ‘noinline’ [-Wattributes]
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); }
^
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:680:44: note: previous definition of ‘void pybind11::pybind11_fail(const char*)’ was here
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); }
^~~~~~~~~~~~~
[ 92%] Building CXX object examples/CMakeFiles/example_carma.dir/automatic_conversion.cpp.o
In file included from /home/mizkulg/carma/third_party/pybind11/include/pybind11/buffer_info.h:12,
from /home/mizkulg/carma/include/carma/carma/converters.h:22,
from /home/mizkulg/carma/include/carma/carma/arraystore.h:1,
from /home/mizkulg/carma/include/carma/carma.h:1,
from /home/mizkulg/carma/examples/automatic_conversion.h:2,
from /home/mizkulg/carma/examples/automatic_conversion.cpp:1:
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:681:83: warning: inline declaration of ‘void pybind11::pybind11_fail(const string&)’ follows declaration with attribute ‘noinline’ [-Wattributes]
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); }
^
/home/mizkulg/carma/third_party/pybind11/include/pybind11/detail/common.h:680:44: note: previous definition of ‘void pybind11::pybind11_fail(const char*)’ was here
[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); }
^~~~~~~~~~~~~
[100%] Linking CXX shared module example_carma.cpython-37m-x86_64-linux-gnu.so
[100%] Built target example_carma
Install the project...
-- Install configuration: "Debug"
-- Installing: /home/mizkulg/carma/build/./include
-- Installing: /home/mizkulg/carma/build/./include/carma
-- Installing: /home/mizkulg/carma/build/./include/carma/carma
-- Installing: /home/mizkulg/carma/build/./include/carma/carma/nparray.h
-- Installing: /home/mizkulg/carma/build/./include/carma/carma/converters.h
-- Installing: /home/mizkulg/carma/build/./include/carma/carma/arraystore.h
-- Installing: /home/mizkulg/carma/build/./include/carma/carma/utils.h
-- Installing: /home/mizkulg/carma/build/./include/carma/carma.h
-- Installing: /home/mizkulg/carma/build/tests/test_arr_to_mat.py
-- Installing: /home/mizkulg/carma/build/tests/test_arraystore.py
-- Installing: /home/mizkulg/carma/build/tests/test_loading.py
-- Installing: /home/mizkulg/carma/build/tests/test_mat_to_arr.py
-- Installing: /home/mizkulg/carma/build/tests/test_nparray.py
-- Installing: /home/mizkulg/carma/build/tests/test_type_caster.py
-- Installing: /home/mizkulg/carma/build/examples/carma_examples.py
~/carma/build master ❯ ctest 1m 23s Py pybind 16:06:38
Test project /home/mizkulg/carma/build
Start 1: pytest
1/2 Test #1: pytest ...........................***Failed 0.08 sec
Start 2: example
2/2 Test #2: example ..........................***Failed 0.05 sec
0% tests passed, 2 tests failed out of 2
Total Test time (real) = 0.13 sec
The following tests FAILED:
1 - pytest (Failed)
2 - example (Failed)
Errors while running CTest
stormshawn commented
Yea I tried with backticks beforehand but I get the following error:
c++ -O3 -Wall -shared -std=c++14 -fPIC -larmadillo `python3 -m pybind11 --includes` carma.cpp -o example `python3-config --extension-suffix`
c++: error: .cpython-37m-x86_64-linux-gnu.so: No such file or directory
RUrlus commented
Yea I tried with backticks beforehand but I get the following error:
c++ -O3 -Wall -shared -std=c++14 -fPIC -larmadillo `python3 -m pybind11 --includes` carma.cpp -o example `python3-config --extension-suffix` c++: error: .cpython-37m-x86_64-linux-gnu.so: No such file or directory
Ah I see, there shouldn't be a space between example and the extension
-o example`python3-config --extension-suffix`
RUrlus commented
Regarding the test issue, can you run pytest in the build directory? I looks like the tests are not running at all rather than some tests failing
stormshawn commented
Wow ok that space was the problem. Could you update the code on this page https://carma.readthedocs.io/en/latest/basic_usage.html so no one else makes this error? Thanks for spotting the error.
Pytest:
============================================= test session starts =============================================
platform linux -- Python 3.7.3, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/shawn/carma/build
collected 95 items
tests/test_arr_to_mat.py ......................... [ 26%]
tests/test_arraystore.py ............. [ 40%]
tests/test_loading.py . [ 41%]
tests/test_mat_to_arr.py ................................. [ 75%]
tests/test_nparray.py ....... [ 83%]
tests/test_type_caster.py ................ [100%]
============================================= 95 passed in 0.28s ==============================================