Repackage pynac as a pip-installable package
Closed this issue · 85 comments
Pynac has a compile-time dependency on the Python library
but is not installed using Python package infrastructure.
This is problematic because Python users cannot install it
using standard Python tools - for example for testing
different Python versions.
It would make sense for src/sage/libs/pynac/pynac.pxd and pynac_wrap.h to be shipped with this Python package, introducing a dependency on Cython. (The only thing blocking this is from sage.libs.gmp.types cimport mpz_t, mpq_t, mpz_ptr, mpq_ptr - but that can be fixed by some minimal cut&paste.)
(The other parts of src/sage/libs/pynac, constant.pxd, constant.pyx, pynac.pyx use various imports from sage and must remain in sage.libs.pynac)
Unfortunately, there is an unrelated project that has claimed the name "Pynac" on pypi.org: https://pypi.org/project/Pynac
Depends on #30446
Depends on #31064
Upstream: Reported upstream. No feedback yet.
CC: @rwst @slel @tobiasdiez @williamstein @kiwifb @tscrim @kliem @videlec @embray
Component: packages: standard
Keywords: pynac, sd110, sd111
Reviewer: François Bissey
Issue created by migration from https://trac.sagemath.org/ticket/30534
Changed keywords from none to pynac
Description changed:
---
+++
@@ -1,3 +1,10 @@
-pynac has a compile-time dependency on the python library but is not installed using Python package infrastructure. This is problematic because Python users cannot install it using standard Python tools - for example for testing different python versions.
+Pynac has a compile-time dependency on the Python library
+but is not installed using Python package infrastructure.
+This is problematic because Python users cannot install it
+using standard Python tools - for example for testing
+different Python versions.
+
+- [Pynac home](http://pynac.org)
+- [Pynac repo](https://github.com/pynac/pynac)
Cc-ing Ralf Stephan who maintains Pynac.
Description changed:
---
+++
@@ -7,4 +7,8 @@
- [Pynac home](http://pynac.org)
- [Pynac repo](https://github.com/pynac/pynac)
+It would make sense for `src/sage/libs/pynac/pynac.pxd` and `pynac_wrap.h` to be shipped with this Python package, introducing a dependency on Cython. (The only thing blocking this is `from sage.libs.gmp.types cimport mpz_t, mpq_t, mpz_ptr, mpq_ptr` - but that can be fixed by some minimal cut&paste.)
+(The other parts of `src/sage/libs/pynac`, `constant.pxd`, `constant.pyx`, `pynac.pyx` use various imports from sage and must remain in `sage.libs.pynac`)
+
+Description changed:
---
+++
@@ -12,3 +12,6 @@
(The other parts of `src/sage/libs/pynac`, `constant.pxd`, `constant.pyx`, `pynac.pyx` use various imports from sage and must remain in `sage.libs.pynac`)
+Unfortunately, there is an unrelated project that has claimed the name "Pynac" on pypi.org: https://pypi.org/project/Pynac
+
+Unfortunately, there is an unrelated project
that has claimed the name "Pynac" on pypi.org:
Collision between
- Pynac = "Py-Dynac" = Python bindings for the Dynac particle tracking code
- Pynac = "Py-GiNaC" = Python derivative of the C++ library GiNaC
Possible solutions:
- rename these projects to Py-Dynac and Py-GiNaC, or Pydynac and Pyginac
- no renaming but use the name pyginac on PyPI for "Py-GiNaC"
Note:
- "Py-Dynac" got the pynac name on PyPI
- "Py-GiNaC" got the pynac.org domain name
Regardless of how we handle getting "Py-GiNaC" on PyPI,
a disambiguation note in both "Pynac" projects would be
nice, with links to each other.
I will propose changes via pull requests or other
to add such a note in relevant places.
Or use pynac-ginac as the PyPI name (more discoverable?).
We briefly discussed it at sd110.
From William Stein (CoCalc) to Everyone: (1:37 PM)
py-ginac is pretty good.
Changed keywords from pynac to pynac, sd110
Replying to @fchapoton:
There is already some
py_ginacsomewhere :
Thanks for the pointer. This appears to be an active project.
From https://sourceforge.net/projects/pyginac.moebinv.p/:
"This is a refresh and maintained version of the previous stalled project https://sourceforge.net/projects/pyginac/ It works with the current version of GiNaC."
The source (git) is at https://sourceforge.net/p/moebinv/pyginac/code/ci/master/tree/
It provides the Python packages cginac and ginac.
It calls itself "pyGiNaC" but it does not define a distribution of this name. It must be built using scons and is not pip-installable - so it also does not claim a project name on PyPI.
Upstream: Reported upstream. No feedback yet.
Changed keywords from pynac, sd110 to pynac, sd110, sd111
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
Branch pushed to git repo; I updated commit sha1. New commits:
44e4dd8 | Fixup |
Commit: 44e4dd8
Author: Matthias Koeppe, ...
The new package installs correctly. sage.libs.pynac needs more work.
I think I'll need some help here from people who know about import and cimport in Cython...
With what aspects? Just when to use one versus the other?
I am getting runtime linker errors like the following on this branch:
[dochtml] import sage.libs.pynac.pynac # initialize pynac before .ring
[dochtml] ImportError: dlopen(/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so, 2): Symbol not found: __ZN5GiNaC12library_initD1Ev
[dochtml] Referenced from: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so
[dochtml] Expected in: flat namespace
[dochtml] in /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-darwin.so
That is beyond my knowledge as it has something to do with how the packages are linked. Perhaps looking at how p_group_cohomology does its setup might have a clue?
Erik, do you know about these things or know a good person to ask?
The double leading underscore is very strange. If you remove one it becomes simply
$ c++filt -n _ZN5GiNaC12library_initD1Ev
GiNaC::library_init::~library_init()
but with two, it is nothing. Some naming or interface is not set right.
First of all let me remark that I wasn't even able to build it at all until I manually removed all files in build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.
Until then there was still the old pynac_wrap.h dangling around and it did not clean itself.
And something isn't right about pynac/setup.py:
(sage-sh) kliem@cofio:sage$ python
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ginac.pynac import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ginac.pynac'
>>> from ginac.libpynac import *
Segmentation fault
I even get segementation fault in ipython when it tries to load the modules of ginac.
This part in numeric.cpp leads to a segmentation fault when importing the module:
3321 #ifdef PYNAC_HAVE_LIBGIAC
3322 giac::gen* numeric::to_giacgen(giac::context* cptr) const
3323 {
3324 if (t == LONG)
3325 return new giac::gen(v._long);
3326 +---- 7 lines: if (t == MPZ) {-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3333 +---- 10 lines: if (t == MPQ) {-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3343 else
3344 return nullptr;
3345 }
3346 #endif
giac might not be linked correctly, but I don't know. Also ./bootstrap is missing from setup.py. Just running ./configure isn't working.
Also I would guess that you need to include the header files as depends=....
What is definitely missing is gmptypes. Also it seems weird that you name the module ginac.libpynac but then you cimport from ginac.pynac import * in pynac.pyx. This isn't defined.
Anyway, if I know just ./bootstrap; make pynac and then include the folder ginac, then I'm back to your orignal problem.
First I got:
ImportError: /home/mi/kliem/Applications/pynac/ginac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZTVN5GiNaC9containerISt6vectorEE
This I made go away by declaring things static in pynac_wrap.h. Then I get the original error about library_init.
The class library_init is defined in ex.h. However, the methods are then defined in utils.cpp.
But even if I change this, the error doesn't go ayay. By now, I don't understand anything anymore.
I noticed while building the package that giac is automagically detected and used. Current sage package explicitly disable it. I remember chatting with Ralph when he made it available and that it was currently broken. So, first thing will be to disable it.
See pynac/pynac#372 "configure: Change default to --with-giac=no"
I pulled on your python_package branch, I guess I should have used the attached tarball.
I've just the branch into python_package and will also prepare a new tarball
Replying to @kliem:
Also it seems weird that you name the module
ginac.libpynacbut then you cimportfrom ginac.pynac import *inpynac.pyx. This isn't defined.
Thanks, renamed now
OK, so pulling from your branch and building with my system packages and tools on gentoo with python setup.py install --prefix=$someprefix I cannot see anything obviously wrong with what is installed. ldd -r pynac.cpython-39-x86_64-linux-gnu.so just report what I am expecting to be missing, that is symbols from python itself.
I am not currently setup for compiling on OS X, which may have trickier issue at linking.
Replying to @kliem:
Also
./bootstrapis missing fromsetup.py. Just running./configureisn't working.
I've now added some instructions to the README
https://github.com/kliem/pynac/tree/python_package
I did some experiments. Not all changes in this branch are necessary to get things to work. In particular, I did many experiments trying to avoid this segmentation fault, but I couldn't find something. It's ipython specific. Sage just complains about linkage.
As reported above I get a weird segmentation fault, whenever I import ginac.pynac.
It goes away magically, if I import ginac.gmptypes beforehand. I don't know why.
Getting sage to start still does not work, because
ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE
I don't know why it would be undefined. It is sage specific. In an Ipython session it is fine even if I add something that uses it to `ginac/pynac.pyx'
+cdef py_gcd(n, k):
+ return n*k
+
+
+def init_function_table():
+ py_funcs.py_gcd = &py_gcd
+If I pip install this, I can run this function just fine in ipython:
(sage-sh) kliem@cofio:pynac$ ipython
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from ginac import gmptypes
In [2]: from ginac.pynac import *
In [3]: init_function_table()
What is even more strange about this segementation fault is that in sage on develop branch I can run import os beforehand to fix this. This does not work in ipython.
Trying to build this branch from vanilla sage, pynac building fails miserably, unlike my simple build from from your git fork. I could import pynac.ginac when build from your git repo, but here I get
Error compiling Cython file:
------------------------------------------------------------
...
# (at your option) any later version.
# http://www.gnu.org/licenses/
#*****************************************************************************
from cpython cimport *
from ginac.pynac cimport *
^
------------------------------------------------------------
ginac/pynac.pyx:18:0: 'ginac/pynac.pxd' not found
Error compiling Cython file:
------------------------------------------------------------
...
defined by GiNaC. This allows us to prevent collisions with C++ level
special functions when a user asks to construct a symbolic function
with the same name.
"""
global GINAC_FN_SERIAL
GINAC_FN_SERIAL = g_registered_functions().size()
^
------------------------------------------------------------
ginac/pynac.pyx:30:22: undeclared name not builtin: g_registered_functions
Compiling ginac/pynac.pyx because it changed.
[1/1] Cythonizing ginac/pynac.pyx
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
cea4cd5 | ci-cygwin-standard.yml: More stages, continue-on-error: true |
cf31b79 | Fixup after cherry-pick |
073124c | Merge branch 't/31084/makefile__add__ptest__targets_that_do_not_depend_on_the_docbuild' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq |
8769bd6 | .github/workflows/ci-cygwin-*.yml: Separate docbuild and ptest |
3826222 | Fix up stage iv |
598b0c2 | local-cygwin-choco: Do not pass through PATH, use full pathname of choco instead |
d65299c | Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq |
a5e4051 | Merge branch 't/30944/tox__improve_local_sudo_ubuntu_standard' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq |
42f4458 | Merge tag '9.3.beta7' into t/31064/ci_cygwin__yml__adjust_to_new_script_packages__bootstrap___prereq |
f24d646 | Merge #31064 |
I have pushed a few more changes to the pynac branch and have set up testing against this ticket - running at https://github.com/mkoeppe/pynac/runs/2010476345
Yes, same error here. ginac/pynax.pdx is missing from the configure file now.
After adding it, the build runs fine. Sage doens't start with the same stupid error:
ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE
The segmentation fault in ipython went away for an ImportError:
ImportError: /srv/public/kliem/sage/local/lib/python3.7/site-packages/ginac/pynac.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN4giac25i_lex_is_strictly_greaterERKNS_7index_mES2_
Replying to @kiwifb:
Trying to build this branch from vanilla sage, pynac building fails miserably, unlike my simple build from from your git fork. I could import pynac.ginac when build from your git repo, but here I get
Error compiling Cython file: ------------------------------------------------------------ ... # (at your option) any later version. # http://www.gnu.org/licenses/ #***************************************************************************** from cpython cimport * from ginac.pynac cimport * ^ ------------------------------------------------------------ ginac/pynac.pyx:18:0: 'ginac/pynac.pxd' not found Error compiling Cython file: ------------------------------------------------------------ ... defined by GiNaC. This allows us to prevent collisions with C++ level special functions when a user asks to construct a symbolic function with the same name. """ global GINAC_FN_SERIAL GINAC_FN_SERIAL = g_registered_functions().size() ^ ------------------------------------------------------------ ginac/pynac.pyx:30:22: undeclared name not builtin: g_registered_functions Compiling ginac/pynac.pyx because it changed. [1/1] Cythonizing ginac/pynac.pyx
Replying to @kliem:
First of all let me remark that I wasn't even able to build it at all until I manually removed all files in
build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.Until then there was still the old
pynac_wrap.hdangling around and it did not clean itself.
Has anyone experienced this? This seems to be a bug in our build system?
Replying to @kliem:
Yes, same error here.
ginac/pynax.pdxis missing from the configure file now.
pynac.pxd is generated by setup.py from pynac.pxd.in via the configure script.
Installation of the new package can be seen to succeed in https://github.com/mkoeppe/pynac/runs/2010476345
Replying to @kliem:
Replying to @kliem:
First of all let me remark that I wasn't even able to build it at all until I manually removed all files in
build/pkgs/sagelib/src/build/cythonized/sage/libs/pynac/.Until then there was still the old
pynac_wrap.hdangling around and it did not clean itself.Has anyone experienced this? This seems to be a bug in our build system?
Yes, quite possible that the install cleaner only handles Python modules and extensions but not package data.
Branch pushed to git repo; I updated commit sha1. New commits:
c0805f5 | build/pkgs/pynac: Use 9c89d8bbc723456b2d94efb78ca1df1820694ff0 |
Just packaged up current HEAD of my pynac python_package branch.
Attachment: py-ginac-0.7.27.tar.gz
Branch pushed to git repo; I updated commit sha1. New commits:
93fe11f | build/pkgs/pynac: Use 5701911df8f00c99bc7c7ed676360729200fb726 |
(and another time.)
This is fixed with the new attachment. Before there was a version where ginac/pynac.pxd was missing from configure for some reason.
Replying to @mkoeppe:
Replying to @kliem:
Yes, same error here.
ginac/pynax.pdxis missing from the configure file now.
pynac.pxdis generated bysetup.pyfrompynac.pxd.invia theconfigurescript.
Installation of the new package can be seen to succeed in https://github.com/mkoeppe/pynac/runs/2010476345
With the new branch I can now do from ginac.pynac import * without an error raising in ipython. The ImportError when loading sage itself still persists.
So maybe this module needs to include something on runtime and this doesn't work. So from ginac.pynac import * works fine, but the cimport doesn't.
In my experiments, the os.path.dirname(__file__) in the setup file didn't do anything. Maybe the include_dirs are set up wrong?
OK, tried with all the new stuff. Stopped at the documentation again but different symbol than in comment:23
[dochtml] ImportError: /home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZN5GiNaC8py_funcsE
which is
$ c++filt -n _ZN5GiNaC8py_funcsE
GiNaC::py_funcs
Running ldd, removing python calls and going through c++filt -n the following symbols are undefined and should be replaced by calls since you cannot link to the python module
ldd -r /home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so | grep _Z | c++filt -n
undefined symbol: GiNaC::py_funcs (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: vtable for GiNaC::container<std::vector> (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::numeric (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::symbol (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::library_init::~library_init() (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::container<std::vector>::tinfo_static (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: vtable for GiNaC::constant (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::I (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::_num0_bp (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::pseries (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: typeinfo for GiNaC::basic (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&, GiNaC::ex const&) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::library_init::library_init() (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_Float(_object*) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, std::vector<GiNaC::ex, std::allocator<GiNaC::ex> > const&, bool) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_I(_object*) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: std::vector<GiNaC::ex, std::allocator<GiNaC::ex> >::~vector() (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::relational::decide() const (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&, GiNaC::ex const&, GiNaC::ex const&) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::sorted_op(unsigned long) const (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::construct_from_basic(GiNaC::basic const&) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::pseries::is_terminating() const (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::registered_functions() (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::compare(GiNaC::ex const&) const (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::numeric::to_pyobject() const (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::relational::the_operator() const (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::function::function(unsigned int, GiNaC::ex const&) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::ex::construct_from_int(int) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::constant::constant() (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::basic::hold() const (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: ginac_pyinit_Integer(_object*) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
undefined symbol: GiNaC::basic::operator=(GiNaC::basic const&) (/home/fbissey/sandbox/git-fork/sage/local/lib/python3.9/site-packages/sage/libs/pynac/pynac.cpython-39-x86_64-linux-gnu.so)
Looks like trying to pull in everything to sage.libs.pynac by from ginac.pynac cimport * is a mistake and instead all modules that need pynac should cimport more specifically from ginac.pynac
Replying to @mkoeppe:
Looks like trying to pull in everything to
sage.libs.pynacbyfrom ginac.pynac cimport *is a mistake and instead all modules that need pynac shouldcimportmore specifically fromginac.pynac
But why does
sage: cython('''
....: from sage.libs.pynac.pynac cimport *
....: ''')
works just fine on develop? I would say that things should work just fine, once everything is set up correctly. However, I don't know how to resolve the problems as I'm currently not even able to cimport from my own little two files project.
I might have found the problem:
In setup.py:
- package_data = {'ginac': ['*.pxd', '*.h', '*.hpp']},
+ package_data = {'ginac': ['*.pxd', '*.h', '*.hpp', '*.cpp']},This will place the .cpp files in the correct directory. We need them, because when we cimport we recompile some of them. When the .cpp files are not there, we get the missing symbols.
After doing this, I get a new error:
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/libs/pynac/constant.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/libs/pynac/constant.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/ring.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/ring.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/function.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/function.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/libs/pynac/pynac.o -L/srv/public/kliem/sage/local/lib -lgsl -lflint -lgmp -lm -lopenblas -lginac -o build/lib.linux-x86_64-3.7/sage/libs/pynac/pynac.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,/srv/public/kliem/sage/local/lib -L/srv/public/kliem/sage/local/lib -Wl,-rpath,/srv/public/kliem/sage/local/lib -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/build/cythonized/sage/symbolic/expression.o -lflint -lgmp -lginac -o build/lib.linux-x86_64-3.7/sage/symbolic/expression.cpython-37m-x86_64-linux-gnu.so -L/srv/public/kliem/sage/local/lib -lfactory -lntl -lgmp -lomalloc -lsingular_resources -lpari
[sagelib-9.3.beta7] /usr/bin/ld: cannot find -lginac
[sagelib-9.3.beta7] collect2: error: ld returned 1 exit status
[sagelib-9.3.beta7] error: command 'g++' failed with exit status 1
But I think, this might be easier to resolve.
This can't be the right fix. The .cpp files should not be installed.
Where's that -lginac coming from??
Branch pushed to git repo; I updated commit sha1. New commits:
b4da02f | cimport directly from ginac.pynac |
Same problem. Same undefined symbol.
Replying to @sagetrac-git:
Branch pushed to git repo; I updated commit sha1. New commits:
b4da02f | cimport directly from ginac.pynac |
Replying to @mkoeppe:
This can't be the right fix. The .cpp files should not be installed.
When you cimport you recompile .pxd file. And by this it is trying to recompile everything that was included there. If the corresponding .cpp files aren't found, there is a problem with that.
If we don't want to recompile everything, then we should put the cdef extern from into pynac.pyx and only do the type definitions in pynax.pxd.
Replying to @kliem:
we should put the
cdef extern fromintopynac.pyxand only do the type definitions inpynax.pxd.
I don't know about this mechanism, but it sounds like what I am looking for. Could you elaborate on this or push a change to the branch?
I'll check, if that fixes things.
I don't know, if that will work. I get many errors and by no means do I know how to fix them.
E.g.
[pynac-0.7.28.p18] gcc -DNDEBUG -g -fwrapv -O2 -Wall -march=native -O3 -g -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iginac -I./ginac -I. -I. -I/srv/public/kliem/sage/local/lib/python3.7/site-packages/pip/_vendor/pep517 -I/srv/public/kliem/sage/local -I/usr/lib/python37.zip -I/usr/lib/python3.7 -I/usr/lib/python3.7/lib-dynload -I/srv/public/kliem/sage/local/lib/python3.7/site-packages -I/srv/public/kliem/sage/local/include -I/usr/include/python3.7m -c ginac/pynac.cpp -o build/temp.linux-x86_64-3.7/ginac/pynac.o -std=c++11 -DSING_NDEBUG -DOM_NDEBUG -DSING_NDEBUG -DOM_NDEBUG -I/srv/public/kliem/sage/local/include
[pynac-0.7.28.p18] ginac/pynac.cpp:1875:50: error: no declaration matches '__pyx_t_5ginac_5pynac_GFunctionOptVector& GiNaC::function::registered_functions()'
[pynac-0.7.28.p18] static __pyx_t_5ginac_5pynac_GFunctionOptVector &GiNaC::function::registered_functions(void); /*proto*/
[pynac-0.7.28.p18] ^~~~~
[pynac-0.7.28.p18] In file included from ginac/ginac.h:54,
[pynac-0.7.28.p18] from ginac/pynac_wrap.h:13,
[pynac-0.7.28.p18] from ginac/pynac.cpp:786:
[pynac-0.7.28.p18] ginac/function.h:409:41: note: candidate is: 'static std::vector<GiNaC::function_options>& GiNaC::function::registered_functions()'
[pynac-0.7.28.p18] static std::vector<function_options> & registered_functions();
[pynac-0.7.28.p18] ^~~~~~~~~~~~~~~~~~~~
[pynac-0.7.28.p18] ginac/function.h:340:7: note: 'class GiNaC::function' defined here
[pynac-0.7.28.p18] class function : public exprseq
I don't even know if it is possible to import a cppclass in the .pyx part and to define in in the header then.
This option is definitely not mentioned here: https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html
The problem with this entire setup is that ginac was never properly compiled and linked as a C++ module. I have the feeling that you are trying to achieve something that cython doesn't want you to do.
- If you have a C++ library you can create a cython header for this and directly expose the cpp classes via that header (e.g. you can create a pip installable GMP header that uses an existing GMP installation).
- If you have some C++ files and you want to expose them directly by a cython wrapper, but the header of this wrapper will recompile your files and thus every module cimporting from this wrapper needs to be able to recompile this as well.
- You can create a proper wrapper class of a cpp class. This class can be used by other modules without recompiling the actual cpp class.
This is what the picture looks like to me.
I digged into a bit. Not that I know a solution now, but at least I understand somehow what is going on.
The question is really what we want. cdef extern from in a .pxd-file assumes that every file cimporting this is linked against the same library or has access to the same sources. I don't think we can change this. So the way this is we must either compile ginac as a library and link against it or ship the sources (the later being really awful, because we have plenty of duplications and it takes forever to compile).
You can use cdef extern in the pyx however and expose the definitions in the header. This seems a very clean way of doing it, but there are tons of obstructions along the way. E.g. the functions must be declared static: https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#implementing-functions-in-c
And apparently with a cppclass it works even in the header? (Different project, but there it seemed to work.)
I'm confused. I will see, if I can find out, what is causing the problem.
Thanks for looking into this!
Changed author from Matthias Koeppe, ... to none
Reviewer: François Bissey
Sure, let's try the other way.
Removing the branch to avoid confusion in the future.
Changed branch from u/mkoeppe/repackage_pynac_as_a_pip_installable_package to none