Fail to `#include <git2.h>` in virtualenv
Graham42 opened this issue · 12 comments
Install process works in a virtualenv but when I say import pygit2 I get a stack trace telling me it can't import git2.h
With python 3:
<path/to/my/virtualenv>/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/__pycache__/_cffi__x4f8db466x390834cc.c:185:18: fatal error: git2.h: No such file or directory
#include <git2.h>
^
compilation terminated.
...
Same thing with python 2:
<path/to/my/virtualenv>/lib/python2.7/site-packages/pygit2-0.21.3-py2.7-linux-x86_64.egg/pygit2/__pycache__/_cffi__x245ec18bx390834cc.c:185:18: fatal error: git2.h: No such file or directory
#include <git2.h>
^
compilation terminated.
...
this issue is over my head, but after some digging, I can get it a little further before it breaks
- install libgit2 and pygit2 in a virtualenv
- export the includes dir:
(venv)$ export C_INCLUDE_PATH=$VIRTUAL_ENV/include (venv)$ python -c "import pygit2"fails with a new error:
/usr/bin/ld: cannot find -lgit2
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
File "/usr/lib64/python3.4/distutils/unixccompiler.py", line 194, in link
self.spawn(linker + ld_args)
File "/usr/lib64/python3.4/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/lib64/python3.4/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib64/python3.4/distutils/spawn.py", line 162, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/ffiplatform.py", line 48, in _build
dist.run_command('build_ext')
File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib64/python3.4/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/usr/lib64/python3.4/distutils/command/build_ext.py", line 448, in build_extensions
self.build_extension(ext)
File "/usr/lib64/python3.4/distutils/command/build_ext.py", line 535, in build_extension
target_lang=language)
File "/usr/lib64/python3.4/distutils/ccompiler.py", line 717, in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
File "/usr/lib64/python3.4/distutils/unixccompiler.py", line 196, in link
raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/__init__.py", line 35, in <module>
from .config import Config
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/config.py", line 32, in <module>
from .errors import check_error
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/errors.py", line 29, in <module>
from .ffi import ffi, C
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/ffi.py", line 59, in <module>
include_dirs=include_dirs, library_dirs=library_dirs)
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/api.py", line 340, in verify
lib = self.verifier.load_library()
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/verifier.py", line 74, in load_library
self._compile_module()
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/verifier.py", line 139, in _compile_module
outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/ffiplatform.py", line 25, in compile
outputfilename = _build(tmpdir, ext)
File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/ffiplatform.py", line 51, in _build
raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: LinkError: command 'gcc' failed with exit status 1
edit: http://stackoverflow.com/questions/5329638/usr-bin-ld-cannot-find makes me think an argument needs to be given in the actual g++ command that cffi is generating. Can we do that with an environment variable or something?
@Graham42 after skimming some pygit2 code I got it to work. It is required to have LIBGIT2 exported as an environment variable at run time, not just when installing.
(venv)$ export LIBGIT2=$VIRTUAL_ENV
(venv)$ python -c "import pygit2"
(venv)$ no errors.
This is a little bit annoying, as previously (with v0.20.x, I think) exporting $LIBGIT2 was only required at build time, and then things just worked.
I am looking into this
When installing pygit2 an extension is generated by cffi with a name like _cffi__xc9c1b883xa706f225.so
What happens is that at run time cffi does not find this extension, so it tries to generate it again, and then you get the build error from the original comment, since the include dirs are not properly defined.
The name of the extension module is a hash from a number of sources including the parameters passed to the cffi verify method, https://github.com/libgit2/pygit2/blob/master/pygit2/ffi.py#L52
In other words, for the name to be the same, so cffi finds the extension, the parameters passed to verify must be the same. That's solution number (1), to be sure, one way or another, that verify is called with the same parameters.
Solution number (2) is to use modulename, https://cffi.readthedocs.org/en/release-0.8/#reference-verifier but there is a caveat, https://cffi.readthedocs.org/en/release-0.8/#warning-about-modulename
After some testing I have not found any practical caveat, every time I call build it regenerates the cffi extension, even when nothing changes.
So I have made the extension name constant, pygit2_cffi, and this fixes the issue for me. Could you give it a try?
Works! Thanks very much!
After some testing I have not found any practical caveat, every time I call build it regenerates the cffi extension, even when nothing changes.
It never regenerates the extension for me (at least on OSX, which is where I'm working atm), even if the C header changes, so when adding functionality via cffi, you need to remember to clean the cache dir. It's definitely better to have easier deploys, but I just wanted to point out that it is some extra work when developing.
Ok, I'm working on this.
Yep, now I can change decl.h without problems.
excuse me, I got the cffi verification error and don't know where to find help. I run 'python setup.py build' on Mac os , virtualenv , and keep get the cffi verify error. Really appreciate it if you can give me some help. The same error when run pip install pygit2(v0.21.4).
same situation in ubuntu 12.04.
Here's the log:
cffi.ffiplatform.VerificationError: importing '/usr/local/var/lib/pyenv/versions/test_env/build/pygit2/pygit2/pycache/pygit2_cffi_18eab927xbf062fb5.so': dlopen(/usr/local/var/lib/pyenv/versions/test_env/build/pygit2/pygit2/pycache/pygit2_cffi_18eab927xbf062fb5.so, 2): Library not loaded: @rpath/libgit2.0.21.2.dylib
Referenced from: /usr/local/var/lib/pyenv/versions/test_env/build/pygit2/pygit2/pycache/pygit2_cffi_18eab927xbf062fb5.so
Reason: image not found
I just solve the problem, the reason is I didn't deal well with the rpath on mac, on ubuntu it's easy which do what the installation doc says sudo ldconfig, I set the DYLD_LIBRARY_PATH to the libgit2 in virtuanenv , and it was solved. Sorry to bother, I'll leave this comment here to help someone 'dumb' like me.