lerouxrgd/ngt-rs

fatal error: 'NGT/Capi.h' file not found

cjrh opened this issue · 5 comments

cjrh commented

Hi @lerouxrgd

I tried building ngt-rs today and it failed. This is the error I see:

  /home/caleb/tmp/ngt-rs/target/debug/build/ngt-sys-8ce361414492c0ac/out/include/NGT/NGTQ/Capi.h:107:10: fatal error: 'NGT/Capi.h' file not found

I checked, and the file NGT/Capi.h is present:

~/tmp/ngt-rs/target/debug/build/ngt-sys-8ce361414492c0ac/out/include/NGT
$ ls -lah
Permissions Size User  Date Modified Name
.rw-r--r--  6.2k caleb  3 Sep 21:44  SharedMemoryAllocator.h
.rw-r--r--   13k caleb  3 Sep 21:44  Tree.h
.rw-r--r--  1.4k caleb  3 Sep 21:44  Version.h
.rw-r--r--   14k caleb  3 Sep 21:44  ObjectRepository.h
.rw-r--r--   18k caleb  3 Sep 21:44  Node.h
.rw-r--r--  7.7k caleb  3 Sep 21:44  Thread.h
.rw-r--r--  5.8k caleb  3 Sep 21:44  ArrayFile.h
.rw-r--r--   20k caleb  3 Sep 21:44  MmapManagerImpl.hpp
.rw-r--r--  2.6k caleb  3 Sep 21:44  MmapManager.h
.rw-r--r--   17k caleb  3 Sep 21:44  ObjectSpace.h
.rw-r--r--   54k caleb  3 Sep 21:44  Optimizer.h
.rw-r--r--   860 caleb  3 Sep 21:44  MmapManagerException.h
.rw-r--r--  2.4k caleb  3 Sep 21:44  MmapManagerDefs.h
.rw-r--r--   29k caleb  3 Sep 21:44  Clustering.h
.rw-r--r--  7.9k caleb  3 Sep 21:44  Capi.h
.rw-r--r--   56k caleb  3 Sep 21:44  Common.h
.rw-r--r--   30k caleb  3 Sep 21:44  GraphReconstructor.h
.rw-r--r--  149k caleb  3 Sep 21:44  half.hpp
.rw-r--r--   22k caleb  3 Sep 21:44  GraphOptimizer.h
.rw-r--r--  4.1k caleb  3 Sep 21:44  Command.h
.rw-r--r--   40k caleb  3 Sep 21:44  Graph.h
.rw-r--r--   254 caleb  3 Sep 21:44  version_defs.h
.rw-r--r--   36k caleb  3 Sep 21:44  PrimitiveComparator.h
.rw-r--r--   31k caleb  3 Sep 21:44  ObjectSpaceRepository.h
.rw-r--r--   59k caleb  3 Sep 21:44  Index.h
.rw-r--r--  2.6k caleb  3 Sep 21:44  HashBasedBooleanSet.h
.rw-r--r--  1.7k caleb  3 Sep 21:44  defines.h
drwxrwxr-x     - caleb  3 Sep 21:45  NGTQ

It seems to me some kind of path problem. Do you have any suggestions for what I can try to fix this, or is it something that must be changed in ngt-rs?

For completeness, I've attached the full build output.

err.log

cjrh commented

Investigating further, I managed to compile purely the NGT source (in the git submodule) successfully, using the instructions from the NGT repo:

~/tmp/ngt-rs/ngt-sys  ±master|…1 
$ ls
build.rs  Cargo.lock  Cargo.toml  LICENSE  NGT  src  target
~/tmp/ngt-rs/ngt-sys  ±master|…1 
$ cd NGT/
~/tmp/ngt-rs/ngt-sys/NGT  ±(detached)|✔︎ 
$ ls
assets  CMakeLists.txt  lib      python        README.jp  samples  tests  VERSION
bin     data            LICENSE  README-jp.md  README.md  tags     utils  
~/tmp/ngt-rs/ngt-sys/NGT  ±(detached)|✔︎ 
$ ls -lah
Permissions Size User  Date Modified Name
.rw-rw-r--    39 caleb  3 Sep 21:44  .git
drwxrwxr-x     - caleb  3 Sep 21:44  assets
.rw-rw-r--  2.8k caleb  3 Sep 21:44  CMakeLists.txt
.rw-rw-r--   11k caleb  3 Sep 21:44  LICENSE
.rw-rw-r--     7 caleb  3 Sep 21:44  VERSION
.rw-rw-r--   167 caleb  3 Sep 21:44  README.jp
.rw-rw-r--  7.8k caleb  3 Sep 21:44  README.md
.rw-rw-r--    56 caleb  3 Sep 21:44  .gitignore
.rw-rw-r--  8.0k caleb  3 Sep 21:44  README-jp.md
drwxrwxr-x     - caleb  3 Sep 21:44  bin
drwxrwxr-x     - caleb  3 Sep 21:44  lib
drwxrwxr-x     - caleb  3 Sep 21:44  data
drwxrwxr-x     - caleb  3 Sep 21:44  python
drwxrwxr-x     - caleb  3 Sep 21:44  samples
drwxrwxr-x     - caleb  3 Sep 21:44  tests
drwxrwxr-x     - caleb  3 Sep 21:44  utils
.rw-rw-r--  405k caleb  3 Sep 22:16  tags
~/tmp/ngt-rs/ngt-sys/NGT  ±(detached)|✔︎ 
$ mkdir build
~/tmp/ngt-rs/ngt-sys/NGT  ±(detached)|✔︎ 
$ cd build
~/tmp/ngt-rs/ngt-sys/NGT/build  ±(detached)|✔︎ 
$ cmake ..
CMake Deprecation Warning at CMakeLists.txt:4 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- VERSION: 1.14.7
-- CMAKE_BUILD_TYPE: Release
-- CMAKE_BUILD_TYPE_LOWER: release
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/caleb/tmp/ngt-rs/ngt-sys/NGT/build
~/tmp/ngt-rs/ngt-sys/NGT/build  ±(detached)|✔︎ 
$ ls
bin  cmake_install.cmake  CMakeCache.txt  CMakeFiles  lib  Makefile  samples
~/tmp/ngt-rs/ngt-sys/NGT/build  ±(detached)|✔︎ 
$ make
[  2%] Generating command
[  2%] Built target versiondef
[  4%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/ArrayFile.cpp.o
[  6%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/Capi.cpp.o

<snip>

[ 77%] Linking CXX executable ngtqg
[ 77%] Built target ngtqg_exe
[ 80%] Building CXX object samples/hamming-uint8/CMakeFiles/hamming-uint8.dir/hamming-uint8.cpp.o
[ 82%] Linking CXX executable hamming-uint8
[ 82%] Built target hamming-uint8
[ 84%] Building CXX object samples/l2-uint8/CMakeFiles/l2-uint8.dir/l2-uint8.cpp.o
[ 86%] Linking CXX executable l2-uint8
[ 86%] Built target l2-uint8
[ 88%] Building CXX object samples/l2-uint8-range-search/CMakeFiles/l2-uint8-range-search.dir/l2-uint8-range-search.cpp.o
[ 91%] Linking CXX executable l2-uint8-range-search
[ 91%] Built target l2-uint8-range-search
[ 93%] Building CXX object samples/cosine-float/CMakeFiles/cosine-float.dir/cosine-float.cpp.o
[ 95%] Linking CXX executable cosine-float
[ 95%] Built target cosine-float
[ 97%] Building CXX object samples/jaccard-sparse/CMakeFiles/jaccard-sparse.dir/jaccard-sparse.cpp.o
/home/caleb/tmp/ngt-rs/ngt-sys/NGT/samples/jaccard-sparse/jaccard-sparse.cpp: In function ‘void append(NGT::Args&)’:
/home/caleb/tmp/ngt-rs/ngt-sys/NGT/samples/jaccard-sparse/jaccard-sparse.cpp:78:21: warning: unused variable ‘id’ [-Wunused-variable]
   78 |       NGT::ObjectID id = index.append(index.makeSparseObject(object));
      |                     ^~
[100%] Linking CXX executable jaccard-sparse
[100%] Built target jaccard-sparse

So it seems to me that perhaps something needs to change in the build.rs?

cjrh commented

Digging further, I've found a change that makes a difference. In build.rs:

    let bindings = bindgen::Builder::default()
        .clang_arg(format!("-F{}/include", dst.display()))
        .header(format!("{}/include/NGT/NGTQ/Capi.h", dst.display()))
        .generate()
        .expect("Unable to generate bindings");

If I change the clang_arg line to use -I instead of -F then the build works:

    let bindings = bindgen::Builder::default()
        .clang_arg(format!("-I{}/include", dst.display()))
        .header(format!("{}/include/NGT/NGTQ/Capi.h", dst.display()))
        .generate()
        .expect("Unable to generate bindings");

Is there a reason why the -F is being used instead of -I? (I don't know much about compiler flags beyond -I).

cjrh commented

This (-I) seems to have fixed my problem. After a successful cargo build in the ngt-rs root, I see the shared objects;

$ fd --hidden --no-ignore --glob libngt*so*
target/debug/build/ngt-sys-8ce361414492c0ac/out/build/lib/NGT/libngt.so
target/debug/build/ngt-sys-8ce361414492c0ac/out/build/lib/NGT/libngt.so.1
target/debug/build/ngt-sys-8ce361414492c0ac/out/build/lib/NGT/libngt.so.1.14.7
target/debug/build/ngt-sys-8ce361414492c0ac/out/lib/libngt.so
target/debug/build/ngt-sys-8ce361414492c0ac/out/lib/libngt.so.1
target/debug/build/ngt-sys-8ce361414492c0ac/out/lib/libngt.so.1.14.7

The tests pass:

$ LD_LIBRARY_PATH=target/debug/build/ngt-sys-8ce361414492c0ac/out/lib/ cargo test
   Compiling autocfg v1.1.0
   Compiling crossbeam-utils v0.8.11
   
   <snip>
   
   Compiling crossbeam-deque v0.8.2
   Compiling ngt v0.4.3 (/home/caleb/tmp/ngt-rs)
    Finished test [unoptimized + debuginfo] target(s) in 5.46s
     Running unittests src/lib.rs (target/debug/deps/ngt-d99b9189f1bb3463)

running 7 tests
test optim::tests::test_convert_anng_to_onng ... ignored
test optim::tests::test_optimize_anng ... ignored
test index::tests::test_basics ... ok
test index::tests::test_batch ... ok
test index::tests::test_quantize ... ok
test optim::tests::test_refine_anng ... ok
test index::tests::test_multithreaded ... ok

test result: ok. 5 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.30s

   Doc-tests ngt

running 2 tests
test src/lib.rs - (line 12) ... ok
test src/lib.rs - (line 32) ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.35s
lerouxrgd commented

Hi, I've added a CI that runs on Unbuntu and using -F looks fine, do you have a specific setup ?

Besides, I will update NGT to 1.14.8 and I will try to use -I at the same time, as I think it should be fine to use it too.

lerouxrgd commented

I've released in 0.4.4.

  • ❤️1