bacpop/pp-sketchlib

installing with conda

Closed this issue · 19 comments

flass commented

Hi Nick and John,

I'm having trouble installing the pp-sketchlib. Here is what I tried:

1. with conda

I Initially tried with conda, in a new conda environment, running with the default python3.7, I get this error:

Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

  - pp-sketchlib -> python[version='>=2.7,<2.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0']

Your python: python=3.7

So I tried and resolve the conflict by changing my python. I get the same error with python3.6 installed:

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - pp-sketchlib -> python[version='>=2.7,<2.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0']

Your python: python=3.6

and a different one with python3.5 installed:

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - pp-sketchlib -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']

Your python: python=3.5

Note that the link to conda forge provided on the README page related to UnsatisfiableError is not really helping from the point-of-view of the wannabe user; maybe more specific guidance tailored for the installation of pp-sketchlib would be more helpful.

also, please note that the instruction for running conda on the README page is wrong:
it is written:

conda -c bioconda install pp-sketchlib

instead of:

conda install -c bioconda pp-sketchlib

2. local build from source

so then i tried to install the package manually.
For that I used conda to get the latest gcc (GCC 7.5.0) and cmake (Cmake version 3.14.0), and i used the pip provided by anaconda (python3.7 base environment) to get the dependencies pybind11 and hdf5.

when trying to install the lib loocally using the recommended command:

python setup.py install

I got this error:

running install
running bdist_egg
running egg_info
writing pp_sketchlib.egg-info/PKG-INFO
writing dependency_links to pp_sketchlib.egg-info/dependency_links.txt
writing entry points to pp_sketchlib.egg-info/entry_points.txt
writing requirements to pp_sketchlib.egg-info/requires.txt
writing top-level names to pp_sketchlib.egg-info/top_level.txt
reading manifest file 'pp_sketchlib.egg-info/SOURCES.txt'
writing manifest file 'pp_sketchlib.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
CMake Error at CMakeLists.txt:25 (find_package):
  By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "pybind11",
  but CMake did not find one.

  Could not find a package configuration file provided by "pybind11" with any
  of the following names:

    pybind11Config.cmake
    pybind11-config.cmake

  Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
  "pybind11_DIR" to a directory containing one of the above files.  If
  "pybind11" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!
See also "/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
  File "setup.py", line 121, in <module>
    zip_safe=False
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 172, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 158, in call_command
    self.run_command(cmdname)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 54, in run
    self.build_extension(ext)
  File "setup.py", line 82, in build_extension
    subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/nfs/users/nfs_f/fl4/software/pp-sketchlib', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/lib.linux-x86_64-3.7', '-DPYTHON_EXECUTABLE=/lustre/scratch118/infgen/team216/fl4/miniconda3/bin/python', '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.

This issue with Cmake files when pybind11 was installed through pip is widely documented, but I must admit I got stuck there. Online forums suggest building manually that lib, but I fail to find where to go to make that build.

All in all, I would be taking any advice or worked out recipe to install or build the pp-sketchlib!

Thanks for the help!

Florent

johnlees commented

Hi Florent,

Sorry this is causing difficulties. I know that there are likely problems unless your conda setup is squeaky-clean.

For 1), I am currently waiting for a new version to be merged into conda-forge, which should fix this (conda-forge/staged-recipes#11135).

For 2) the best way to do this is to first make a new conda environment with the dependencies (see environment.yml), activate it, then run python setup.py install

flass commented

Hi John,

thank you for your answer and advice. I will try building from source within a conda environment based on the environment.yml file.
Just had a look to it, and I see it does not include specifying the compiler; should I add a dependency request for gcc >=7.2.0?

johnlees commented

Ah, that's can be a bit trickier. In the first instance, I would have a look at the compiler version available on your machine and see if it's a high enough version. If you're on OS X clang should also work, but you can get gcc through homebrew. If you're on the sanger farm, they have higher versions available through module load, but iirc farm5 finally had gcc 7

flass commented

Sanger farm has gcc 7.5.0 by default … and here I just realised I got confused with the Cmake requirement, pp-sketchlib only requires gcc >= 7.2.0, not >= 7.12.0. - so I’m sorted! :-)
Sorry about the unnecessary hassle, and thanks again for the help!

flass commented

So I confirm I satisfied the dependency requirements using the pp_env conda environment as specified through the environment.yml file.
But then, I got an error while running python setup.py install:
please see below:

[ 30%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/reference.cpp.o
/usr/bin/c++  -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -isystem /lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/include -isystem /lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/include/python3.8 -isystem /lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/include/eigen3  -DVERSION_INFO=\"1.2.1\" -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -O3 -DNDEBUG -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden   -fvisibility=hidden -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/reference.cpp.o -c /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:19:0,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:11:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:27:2: error: expected ‘;’ after struct definition
 }
  ^
  ;
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:40:9: error: invalid use of template-name ‘BaseComp’ without an argument list
         BaseComp get_composition() const { return _bases; }
         ^~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:40:9: note: class template argument deduction is only available with -std=c++1z or -std=gnu++1z
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:21:8: note: ‘template<class T> struct BaseComp’ declared here
 struct BaseComp {
        ^~~~~~~~
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:11:0:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:70:9: error: invalid use of template-name ‘BaseComp’ without an argument list
         BaseComp _bases;
         ^~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:70:9: note: class template argument deduction is only available with -std=c++1z or -std=gnu++1z
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:19:0,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:11:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:21:8: note: ‘template<class T> struct BaseComp’ declared here
 struct BaseComp {
        ^~~~~~~~
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:11:0:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp: In member function ‘std::vector<double> Reference::base_composition() const’:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:52:64: error: ‘_bases’ was not declared in this scope
         std::vector<double> base_composition() const { return {_bases.a, _bases.c, _bases.g, _bases.t}; }
                                                                ^~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:52:64: note: suggested alternative: ‘_name’
         std::vector<double> base_composition() const { return {_bases.a, _bases.c, _bases.g, _bases.t}; }
                                                                ^~~~~~
                                                                _name
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:52:102: error: could not convert ‘{<expression error>, <expression error>, <expression error>, <expression error>}’ from ‘<brace-enclosed initializer list>’ to ‘std::vector<double>’
         std::vector<double> base_composition() const { return {_bases.a, _bases.c, _bases.g, _bases.t}; }
                                                                                                      ^
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp: In constructor ‘Reference::Reference(const string&, const std::vector<std::__cxx11::basic_string<char> >&, const std::vector<long unsigned int>&, size_t, bool, uint8_t, bool)’:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:54:5: error: ‘_bases’ was not declared in this scope
     _bases = sequence._bases
     ^~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:54:5: note: suggested alternative: ‘_name’
     _bases = sequence._bases
     ^~~~~~
     _name
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:54:23: error: ‘BaseComp<double> SeqBuf::_bases’ is private within this context
     _bases = sequence._bases
                       ^~~~~~
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:19:0,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:11:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:54:26: note: declared private here
         BaseComp<double> _bases;
                          ^~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:61:9: error: ‘size_sum’ was not declared in this scope
         size_sum += estimated_size;
         ^~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:61:9: note: suggested alternative: ‘size_t’
         size_sum += estimated_size;
         ^~~~~~~~
         size_t
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:63:37: error: ‘size_sum’ was not declared in this scope
     _seq_size = static_cast<size_t>(size_sum / (double)kmer_lengths.size());
                                     ^~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:63:37: note: suggested alternative: ‘size_t’
     _seq_size = static_cast<size_t>(size_sum / (double)kmer_lengths.size());
                                     ^~~~~~~~
                                     size_t
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp: In constructor ‘Reference::Reference(const string&, size_t, size_t, size_t, std::vector<double>)’:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:77:5: error: ‘_bases’ was not declared in this scope
     _bases.a = bases[0];
     ^~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:77:5: note: suggested alternative: ‘bases’
     _bases.a = bases[0];
     ^~~~~~
     bases
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp: At global scope:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:83:8: error: prototype for ‘double Reference::random_match(int)’ does not match any in class ‘Reference’
 double Reference::random_match(const int kmer_len)
        ^~~~~~~~~
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:11:0:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:42:16: error: candidate is: double Reference::random_match(int) const
         double random_match(const int kmer_len) const;
                ^~~~~~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp: In member function ‘double Reference::jaccard_dist(const Reference&, int) const’:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:105:36: error: no matching function for call to ‘Reference::random_match() const’
     double r1 = this->random_match();
                                    ^
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:11:0:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:42:16: note: candidate: double Reference::random_match(int) const
         double random_match(const int kmer_len) const;
                ^~~~~~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:42:16: note:   candidate expects 1 argument, 0 provided
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:106:36: error: no matching function for call to ‘Reference::random_match() const’
     double r2 = query.random_match();
                                    ^
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:11:0:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:42:16: note: candidate: double Reference::random_match(int) const
         double random_match(const int kmer_len) const;
                ^~~~~~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:42:16: note:   candidate expects 1 argument, 0 provided
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:109:70: error: no matching function for call to ‘observed_excess(double&, double&, int)’
     double jaccard = observed_excess(jaccard_obs, jaccard_expected, 1);
                                                                      ^
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:14:0:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/dist.hpp:19:10: note: candidate: template<class T> T observed_excess(T, T, T)
 inline T observed_excess(T obs, T exp, T max) {
          ^~~~~~~~~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/dist.hpp:19:10: note:   template argument deduction/substitution failed:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.cpp:109:70: note:   deduced conflicting types for parameter ‘T’ (‘double’ and ‘int’)
     double jaccard = observed_excess(jaccard_obs, jaccard_expected, 1);
                                                                      ^
CMakeFiles/pp_sketchlib.dir/build.make:108: recipe for target 'CMakeFiles/pp_sketchlib.dir/src/reference.cpp.o' failed
make[2]: *** [CMakeFiles/pp_sketchlib.dir/src/reference.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:19:0,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/api.hpp:17,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/sketchlib_bindings.cpp:16:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:27:2: error: expected ‘;’ after struct definition
 }
  ^
  ;
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:40:9: error: invalid use of template-name ‘BaseComp’ without an argument list
         BaseComp get_composition() const { return _bases; }
         ^~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:40:9: note: class template argument deduction is only available with -std=c++1z or -std=gnu++1z
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:21:8: note: ‘template<class T> struct BaseComp’ declared here
 struct BaseComp {
        ^~~~~~~~
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/api.hpp:17:0,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/sketchlib_bindings.cpp:16:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:70:9: error: invalid use of template-name ‘BaseComp’ without an argument list
         BaseComp _bases;
         ^~~~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:70:9: note: class template argument deduction is only available with -std=c++1z or -std=gnu++1z
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:19:0,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/api.hpp:17,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/sketchlib_bindings.cpp:16:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/seqio.hpp:21:8: note: ‘template<class T> struct BaseComp’ declared here
 struct BaseComp {
        ^~~~~~~~
In file included from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/api.hpp:17:0,
                 from /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/sketchlib_bindings.cpp:16:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp: In member function ‘std::vector<double> Reference::base_composition() const’:
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:52:64: error: ‘_bases’ was not declared in this scope
         std::vector<double> base_composition() const { return {_bases.a, _bases.c, _bases.g, _bases.t}; }
                                                                ^~~~~~
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:52:64: note: suggested alternative: ‘_name’
         std::vector<double> base_composition() const { return {_bases.a, _bases.c, _bases.g, _bases.t}; }
                                                                ^~~~~~
                                                                _name
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/reference.hpp:52:102: error: could not convert ‘{<expression error>, <expression error>, <expression error>, <expression error>}’ from ‘<brace-enclosed initializer list>’ to ‘std::vector<double>’
         std::vector<double> base_composition() const { return {_bases.a, _bases.c, _bases.g, _bases.t}; }
                                                                                                      ^
CMakeFiles/pp_sketchlib.dir/build.make:82: recipe for target 'CMakeFiles/pp_sketchlib.dir/src/sketchlib_bindings.cpp.o' failed
make[2]: *** [CMakeFiles/pp_sketchlib.dir/src/sketchlib_bindings.cpp.o] Error 1
make[2]: Leaving directory '/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.8'
CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/pp_sketchlib.dir/all' failed
make[1]: *** [CMakeFiles/pp_sketchlib.dir/all] Error 2
make[1]: Leaving directory '/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.8'
Makefile:103: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 92, in <module>
    setup(
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 173, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 159, in call_command
    self.run_command(cmdname)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 54, in run
    self.build_extension(ext)
  File "setup.py", line 83, in build_extension
    subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2.

any ideas?

johnlees commented

Is this using the code on the master branch? That has some unfinished code with errors (sorry) which I hadn't intended to release yet. Could you try checking out v1.2.0 git checkout v1.2.0, which should work

flass commented

yes it worked with the release v1.2.0. Thanks!

flass commented

to run the latest PopPUNK I needed to compile pp-sketchlib v1.1.0
(see bacpop/PopPUNK#70).
That did not work, see the error below. (note I made a clean clone of the repo before co mpiling/installing).
I used the same environment in which I successfully built the v1.2.0 (see messages above).

[ 90%] Building CXX object CMakeFiles/pp_sketchlib.dir/src/linear_regression.cpp.o
/usr/bin/c++  -DPYTHON_EXT -Dpp_sketchlib_EXPORTS -I/nfs/users/nfs_f/fl4/software/pp-sketchlib/dlib -I/nfs/users/nfs_f/fl4/software/pp-sketchlib/HighFive/include -I/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/include -I/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/include/python3.8 -isystem /lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/include/eigen3  -DVERSION_INFO=\"1.1.0\" -DMKL_ILP64 -m64 -O3 -DNDEBUG -fPIC -fvisibility=hidden   -D DLIB_NO_GUI_SUPPORT=1 -D DLIB_USE_BLAS=1 -D DLIB_USE_LAPACK=1 -flto -fno-fat-lto-objects -pthread -std=gnu++14 -o CMakeFiles/pp_sketchlib.dir/src/linear_regression.cpp.o -c /nfs/users/nfs_f/fl4/software/pp-sketchlib/src/linear_regression.cpp
/nfs/users/nfs_f/fl4/software/pp-sketchlib/src/linear_regression.cpp:11:10: fatal error: dlib/matrix.h: No such file or directory
 #include <dlib/matrix.h>
          ^~~~~~~~~~~~~~~
compilation terminated.
CMakeFiles/pp_sketchlib.dir/build.make:186: recipe for target 'CMakeFiles/pp_sketchlib.dir/src/linear_regression.cpp.o' failed
make[2]: *** [CMakeFiles/pp_sketchlib.dir/src/linear_regression.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.8'
CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/pp_sketchlib.dir/all' failed
make[1]: *** [CMakeFiles/pp_sketchlib.dir/all] Error 2
make[1]: Leaving directory '/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.8'
Makefile:103: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 89, in <module>
    setup(
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 173, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 159, in call_command
    self.run_command(cmdname)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 54, in run
    self.build_extension(ext)
  File "setup.py", line 80, in build_extension
    subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2.

is there any stable intermediate that allows the use of latest PopPUNK without having the GPU utilities ?

johnlees commented

Sorry again, the dlib submodule was removed, and wouldn't appear in your clone. Can you run git submodule add https://github.com/davisking/dlib and try again (and if needed git submodule add https://github.com/BlueBrain/HighFive but it looks like that is working)

flass commented

hmm, it seems it's not doing anything as the submodules are already indexed:

(pp_env) [fl4@farm5-head1 pp-sketchlib]$ git submodule add https://github.com/davisking/dlib
'dlib' already exists in the index
(pp_env) [fl4@farm5-head1 pp-sketchlib]$ git submodule add https://github.com/BlueBrain/HighFive
'HighFive' already exists in the index

(this being with v1.1.0)

johnlees commented

Is the dlib directory there? You could just run git clone https://github.com/davisking/dlib

flass commented

Sorry I'm not coming back to you with good news 😅

The submodule folders were there, but empty. this brought the code in:

(pp_env) [fl4@farm5-head1 pp-sketchlib]$ git submodule init
Submodule 'HighFive' (https://github.com/BlueBrain/HighFive.git) registered for path 'HighFive'
Submodule 'dlib' (https://github.com/davisking/dlib.git) registered for path 'dlib'
(pp_env) [fl4@farm5-head1 pp-sketchlib]$ git submodule update --recursive
Cloning into '/nfs/users/nfs_f/fl4/software/pp-sketchlib/HighFive'...
Cloning into '/nfs/users/nfs_f/fl4/software/pp-sketchlib/dlib'...
Submodule path 'HighFive': checked out 'af146892cf1769c5488d14a9f1706b72ee25d2c2'
Submodule path 'dlib': checked out 'e82e2ceb2b1d256456214d101b352b786e307e76'

then trying to install pp-sketchlib I got this error:

[ 20%] Linking CXX shared module ../lib.linux-x86_64-3.8/pp_sketchlib.cpython-38-x86_64-linux-gnu.so
/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/bin/cmake -E cmake_link_script CMakeFiles/pp_sketchlib.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -DVERSION_INFO=\"1.1.0\" -DMKL_ILP64 -m64 -O3 -DNDEBUG  -shared  -o ../lib.linux-x86_64-3.8/pp_sketchlib.cpython-38-x86_64-linux-gnu.so CMakeFiles/pp_sketchlib.dir/src/sketchlib_bindings.cpp.o CMakeFiles/pp_sketchlib.dir/src/dist.cpp.o CMakeFiles/pp_sketchlib.dir/src/reference.cpp.o CMakeFiles/pp_sketchlib.dir/src/seqio.cpp.o CMakeFiles/pp_sketchlib.dir/src/countmin.cpp.o CMakeFiles/pp_sketchlib.dir/src/sketch.cpp.o CMakeFiles/pp_sketchlib.dir/src/database.cpp.o CMakeFiles/pp_sketchlib.dir/src/api.cpp.o CMakeFiles/pp_sketchlib.dir/src/linear_regression.cpp.o   -L/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib  -L/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/intel64  -Wl,-rpath,/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib:/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/intel64 -flto -lz -lbz2 -lhdf5_cpp -lhdf5 -lmkl_rt -lm -ldl -pthread
/usr/bin/ld: cannot find -lmkl_rt
collect2: error: ld returned 1 exit status
CMakeFiles/pp_sketchlib.dir/build.make:223: recipe for target '../lib.linux-x86_64-3.8/pp_sketchlib.cpython-38-x86_64-linux-gnu.so' failed
make[2]: *** [../lib.linux-x86_64-3.8/pp_sketchlib.cpython-38-x86_64-linux-gnu.so] Error 1
make[2]: Leaving directory '/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.8'
CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/pp_sketchlib.dir/all' failed
make[1]: *** [CMakeFiles/pp_sketchlib.dir/all] Error 2
make[1]: Leaving directory '/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.8'
Makefile:103: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 89, in <module>
    setup(
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 173, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 159, in call_command
    self.run_command(cmdname)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 54, in run
    self.build_extension(ext)
  File "setup.py", line 80, in build_extension
    subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2.

so I though i should better build those submodule libs:

cd dlib/
python setup.py install
# could not find anything to do with HighFive

then tried again to install pp-sketchlib but no luck as it failed even earlier:

[ 10%] Linking CXX shared module ../lib.linux-x86_64-3.8/pp_sketchlib.cpython-38-x86_64-linux-gnu.so
/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/bin/cmake -E cmake_link_script CMakeFiles/pp_sketchlib.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -DVERSION_INFO=\"1.1.0\" -DMKL_ILP64 -m64 -O3 -DNDEBUG  -shared  -o ../lib.linux-x86_64-3.8/pp_sketchlib.cpython-38-x86_64-linux-gnu.so CMakeFiles/pp_sketchlib.dir/src/sketchlib_bindings.cpp.o CMakeFiles/pp_sketchlib.dir/src/dist.cpp.o CMakeFiles/pp_sketchlib.dir/src/reference.cpp.o CMakeFiles/pp_sketchlib.dir/src/seqio.cpp.o CMakeFiles/pp_sketchlib.dir/src/countmin.cpp.o CMakeFiles/pp_sketchlib.dir/src/sketch.cpp.o CMakeFiles/pp_sketchlib.dir/src/database.cpp.o CMakeFiles/pp_sketchlib.dir/src/api.cpp.o CMakeFiles/pp_sketchlib.dir/src/linear_regression.cpp.o   -L/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib  -L/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/intel64  -Wl,-rpath,/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib:/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/intel64 -flto -lz -lbz2 -lhdf5_cpp -lhdf5 -lmkl_rt -lm -ldl -pthread
/usr/bin/ld: cannot find -lmkl_rt
collect2: error: ld returned 1 exit status
CMakeFiles/pp_sketchlib.dir/build.make:223: recipe for target '../lib.linux-x86_64-3.8/pp_sketchlib.cpython-38-x86_64-linux-gnu.so' failed
make[2]: *** [../lib.linux-x86_64-3.8/pp_sketchlib.cpython-38-x86_64-linux-gnu.so] Error 1
make[2]: Leaving directory '/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.8'
CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/pp_sketchlib.dir/all' failed
make[1]: *** [CMakeFiles/pp_sketchlib.dir/all] Error 2
make[1]: Leaving directory '/nfs/users/nfs_f/fl4/software/pp-sketchlib/build/temp.linux-x86_64-3.8'
Makefile:103: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 89, in <module>
    setup(
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 173, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 159, in call_command
    self.run_command(cmdname)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/command/install_lib.py", line 107, in build
    self.run_command('build_ext')
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 54, in run
    self.build_extension(ext)
  File "setup.py", line 80, in build_extension
    subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
  File "/lustre/scratch118/infgen/team216/fl4/miniconda3/envs/pp_env/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2.
johnlees commented

Ah yes, I forgot the libraries used changed too (this is to get the conda install to work for v1.2.0!). Sorry this is more manual effort than I'm sure you envisaged, but if you edit CMakeLists.txt and change the end of the file from:

if(UNIX AND NOT APPLE)
    target_link_libraries("${TARGET_NAME}" PRIVATE Eigen3::Eigen z bz2 hdf5_cpp hdf5 mkl_rt Threads::Threads m dl)
else()
    target_link_libraries("${TARGET_NAME}" PRIVATE Eigen3::Eigen z bz2 hdf5_cpp hdf5 openblas Threads::Threads m dl)
endif()

to

target_link_libraries("${TARGET_NAME}" PRIVATE Eigen3::Eigen z bz2 hdf5_cpp hdf5 openblas Threads::Threads m dl)

(i.e. get rid of the option using mkl, and just keep the second line with openblas
then get rid of your build directory rm -rf build and try again, it should work!

johnlees commented

Or, if that still doesn't work, replace with the current linker line:

target_link_libraries("${TARGET_NAME}" PRIVATE pybind11::module Eigen3::Eigen 
                                               z hdf5_cpp hdf5 openblas lapack gfortran m dl)
flass commented

OK, given the first solution, pp-sketchlib v1.1.0 built with success, but I got the same error from PopPUNK as reported originally here bacpop/PopPUNK#70:

Traceback (most recent call last):
  File "/nfs/users/nfs_f/fl4/software/PopPUNK/poppunk-runner.py", line 10, in <module>
    main()
  File "/nfs/users/nfs_f/fl4/software/PopPUNK/PopPUNK/__main__.py", line 232, in main
    dbFuncs = setupDBFuncs(args, kmers, args.min_kmer_count)
TypeError: setupDBFuncs() missing 2 required positional arguments: 'use_gpu' and 'gpu_id'

I re-build the pp-sketchlib v1.1.0 with the second solution, again with success, but still got the same PopPUNK error....

johnlees commented

Can you checkout 6b60dc79ae568c53bff41f79230dcdd737dae382 of PopPUNK? That's before any GPU code.

flass commented

using the commit 6b60dc7 i get not GPU-related error, but this one:

PopPUNK: error: unrecognized arguments: --estimated-length 4000000

I suppose it's ahead of the introduction of the option.
so I check the commit network and went for this one 00f4679, which is the tip of the origin/adjusted_distance branch. it got me that far:

Sketching 7628 genomes using 8 thread(s)
Writing sketches to file
PopPUNK (POPulation Partitioning Using Nucleotide Kmers)
        (with backend: sketchlib v1.1.0)
Mode: Creating clusters from assemblies (create_db & fit_model)
Worst random match probability at 13-mers: 0.03
Problem calculating 0.0
Traceback (most recent call last):
  File "/nfs/users/nfs_f/fl4/software/PopPUNK/poppunk-runner.py", line 10, in <module>
    main()
  File "/nfs/users/nfs_f/fl4/software/PopPUNK/PopPUNK/__main__.py", line 275, in main
    adjustment_values = constructDatabase(args.r_files, kmers, sketch_sizes, args.output, args.estimated_length, args.ignore_length, args.threads,
  File "/nfs/users/nfs_f/fl4/software/PopPUNK/PopPUNK/sketchlib.py", line 336, in constructDatabase
    pp_sketchlib.constructDatabase(dbname, names, sequences, klist, sketch_size, min_count, threads)
pp_sketchlib.HDF5Exception: Unable to create file /lustre/scratch118/infgen/team216/fl4/poppunk_7kVc/7kVc//lustre/scratch118/infgen/team216/fl4/poppunk_7kVc/7kVc.h5 (File accessibilty) Unable to open file

which seems to be a bug of file path specification.

the program call was:

/nfs/users/nfs_f/fl4/software/PopPUNK/poppunk-runner.py --easy-run --r-files /lustre/scratch118/infgen/team216/fl4/poppunk_7kVc/7kVibrioCholerae_genome_fasta_list.tab --output /lustre/scratch118/infgen/team216/fl4/poppunk_7kVc/7kVc --threads 8 --plot-fit 5 --min-k 13 --max-k 29 --full-db --microreact --ignore-length --estimated-length 4000000

maybe I should give it a rest and let you get the stable release out, so I avoid running into bugs from intermediate versions?

johnlees commented

I think your install is working ok now. If you run that command, but with --output 7kVc it should work (the output has to be in the current directory, which I realise can be a limitation!).

But I understand if you'd rather wait until I release stable versions before going ahead, this might be a good idea as there are a few enhancements nearly ready too

flass commented

I confirm that I could run PopPUNK with that install; I ran into another bug, but I will report that into another thread for more clarity