Install fewer static libraries
embray opened this issue · 79 comments
As pointed out on sage-devel, by default many SPKGs install static libraries in addition to their associated dynamic libs.
I usually ignored this as harmless, but as Marc pointed out, some of them are quite large, especially libgiac.a. Almost none of these are actually needed either at build or runtime and do not need to be installed by default in SAGE_LOCAL.
I think by default we should configure the relevant packages to not install static libs, or remove them manually if the packages' build system installs them forcibly. This could be overridden with <PKGNAME>_CONFIGURE variables if someone needs the static lib for some reason.
At least the following packages are installing static libs
$ for f in local/var/lib/sage/installed/*; do if grep '\.a\"' "$f" > /dev/null; then echo "$f" | cut -d'/' -f6; fi; done
cddlib-0.94j
cliquer-1.21.p4
e_antic-0.1.3
ecl-16.1.2.p5
eclib-20190909
ecm-7.0.4.p1
fplll-5.2.1
freetype-2.9.1
gap-4.10.2.p1
gc-7.6.4.p0
gf2x-1.2.p0
giac-1.5.0.63.p0
givaro-4.1.1
gsl-2.5
iml-1.0.4p1.p2
libatomic_ops-7.6.2
libgd-2.1.1.1.p1
libpng-1.6.29.p1
lrcalc-1.2.p1
mpc-1.1.0
mpfi-1.5.2
mpfr-4.0.1.p0
mpir-3.0.0-644faf502c56f97d9accd301965fc57d6ec70868.p0
ncurses-6.0.p0
normaliz-3.7.2
ntl-11.3.2
numpy-1.16.1
pari-2.11.1.p2
planarity-3.0.0.5.p0
ppl-1.2.p1
python2-2.7.15.p1
python3-3.7.3.p1
ratpoints-2.1.3.p5
rw-0.7.p0
singular-4.1.1p2.p0
sqlite-3290000
symmetrica-2.0.p11
yasm-1.3.0.p0
zeromq-4.2.5
zlib-1.2.11.p0
In at least a few cases, like libatomic_ops, I believe this is intentional. But in most cases it is probably unnecessary.
Additional notes:
-
cliquer: it already passes
--disable-staticbut it installs a test file intoshare/which happens to end with.abut is not a static lib as far as I can tell. -
ecl: The
.amodules installed with ECL seem to be pretty baked into its build process and I'm not sure if there's an easy way to omit them or if it's even desirable to. -
ecm: apparently we deliberately build it as a static lib by default; not sure why. Need to investigate (performance reasons?)
-
gap: The SmallGrp package also installs some ".a" file that are not static libs; otherwise GAP is not installing any static libs (some things in gap_packages might but I haven't checked yet).
-
mpir: mpir/gmp is linked statically for at least some libraries such as ECM, and possibly some others (GP?).
-
ntl: NTL's build system makes static library installation pretty mandatory--I don't know if there's any good reason for that or not, so for now we can just leave it alone.
-
python2/3: The static lib installation (performed by the
libainstallmake target) is a part of Python's standard installation process and more trouble to remove than it's worth, although I don't think we explicitly need it for anything. -
ratpoints: Has only ever supported being built as a static library, and is statically linked into Sage's ratpoints interface.
-
yasm: If I understand correctly, libyasm contains some special support functions and is intended to be linked statically.
CC: @mezzarobba @dimpase @orlitzky @jhpalmieri @videlec
Component: build
Keywords: static
Author: Erik Bray, Matthias Koeppe
Branch/Commit: c8ccab6
Reviewer: Marc Mezzarobba, John Palmieri
Issue created by migration from https://trac.sagemath.org/ticket/28890
Description changed:
---
+++
@@ -3,3 +3,51 @@
I usually ignored this as harmless, but as Marc pointed out, some of them are quite large, especially libgiac.a. Almost none of these are actually needed either at build or runtime and do not need to be installed by default in SAGE_LOCAL.
I think by default we should configure the relevant packages to *not* install static libs, or remove them manually if the packages' build system installs them forcibly. This could be overridden with `<PKGNAME>_CONFIGURE` variables if someone needs the static lib for some reason.
+
+At least the following packages are installing static libs
+
+```
+$ for f in local/var/lib/sage/installed/*; do if grep '\.a\"' "$f" > /dev/null; then echo "$f" | cut -d'/' -f6; fi; done
+cddlib-0.94j
+cliquer-1.21.p4
+e_antic-0.1.3
+ecl-16.1.2.p5
+eclib-20190909
+ecm-7.0.4.p1
+fplll-5.2.1
+freetype-2.9.1
+gap-4.10.2.p1
+gc-7.6.4.p0
+gf2x-1.2.p0
+giac-1.5.0.63.p0
+givaro-4.1.1
+gsl-2.5
+iml-1.0.4p1.p2
+libatomic_ops-7.6.2
+libgd-2.1.1.1.p1
+libpng-1.6.29.p1
+lrcalc-1.2.p1
+mpc-1.1.0
+mpfi-1.5.2
+mpfr-4.0.1.p0
+mpir-3.0.0-644faf502c56f97d9accd301965fc57d6ec70868.p0
+ncurses-6.0.p0
+normaliz-3.7.2
+ntl-11.3.2
+numpy-1.16.1
+pari-2.11.1.p2
+planarity-3.0.0.5.p0
+ppl-1.2.p1
+python2-2.7.15.p1
+python3-3.7.3.p1
+ratpoints-2.1.3.p5
+rw-0.7.p0
+singular-4.1.1p2.p0
+sqlite-3290000
+symmetrica-2.0.p11
+yasm-1.3.0.p0
+zeromq-4.2.5
+zlib-1.2.11.p0
+```
+
+In at least a few cases, like `libatomic_ops`, I believe this is intentional. But in most cases it is probably unnecessary.Description changed:
---
+++
@@ -51,3 +51,11 @@
```
In at least a few cases, like `libatomic_ops`, I believe this is intentional. But in most cases it is probably unnecessary.
+
+Additional notes:
+
+* cliquer: it already passes `--disable-static` but it installs a test file into `share/` which happens to end with `.a` but is not a static lib as far as I can tell.
+
+* ecm: apparently we deliberately build it as a static lib by default; not sure why. Need to investigate (performance reasons?)
+
+* gap: The SmallGrp package also installs some ".a" file that are not static libs; otherwise GAP is not installing any static libs (some things in gap_packages might but I haven't checked yet).Description changed:
Author: Erik Bray
Branch: u/embray/ticket-28890
Passing --disable-static to all autoconf-compatible packages by default takes care of most of them, narrowing the list down to:
ecl-16.1.2.p5
ecm-7.0.4.p1
libatomic_ops-7.6.2
mpir-3.0.0-644faf502c56f97d9accd301965fc57d6ec70868.p0
ntl-11.3.2
numpy-1.16.1
pari-2.11.1.p2
python2-2.7.15.p1
python3-3.7.3.p1
ratpoints-2.1.3.p5
symmetrica-2.0.p11
yasm-1.3.0.p0
Left libatomic_ops static since I think gc likes to link to it statically for performance reasons, but I need to double-check that it's actually doing that.
The rest I'm not sure about yet.
New commits:
f773cd4 | Trac #28890: Pass --disable-static to autoconf packages by default. |
OK, let me run tests...
the branch looks good, do you want it to be reviewed?
Thanks a lot for taking care of that. (You are crossing an item off the depths of my todo-list!)
I just rebuilt sage from scratch with this branch. With standard packages only and using system packages when possible, the remaining static libraries are
107M ./local/lib/libpari.a
48M ./local/lib/libsymmetrica.a
31M ./local/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a
7,7M ./local/lib/ecl-16.1.2/libcmp.a
7,5M ./local/lib/libbraiding.a
5,4M ./local/lib/libyasm.a
4,5M ./local/lib/ecl-16.1.2/libasdf.a
3,1M ./local/lib/libecm.a
880K ./local/lib/ecl-16.1.2/libdefsystem.a
564K ./local/lib/ecl-16.1.2/libdeflate.a
448K ./local/lib/ecl-16.1.2/libsockets.a
372K ./local/lib/python3.7/site-packages/numpy/core/lib/libnpymath.a
232K ./local/lib/ecl-16.1.2/libecl-help.a
200K ./local/lib/ecl-16.1.2/libprofile.a
184K ./local/lib/ecl-16.1.2/librt.a
176K ./local/lib/libratpoints.a
172K ./local/lib/libhomfly.a
172K ./local/lib/ecl-16.1.2/libecl-cdb.a
140K ./local/lib/ecl-16.1.2/libecl-curl.a
132K ./local/lib/ecl-16.1.2/libserve-event.a
132K ./local/lib/ecl-16.1.2/libql-minitar.a
96K ./local/lib/ecl-16.1.2/libecl-quicklisp.a
72K ./local/lib/ecl-16.1.2/libsb-bsd-sockets.a
32K ./local/lib/libatomic_ops_gpl.a
24K ./local/lib/libatomic_ops.a
So most of the large ones are gone, and those that remain may have a good reason to be there. Like Dima, I'm ready to give this branch positive review if you want.
I want to have a look at a few of the remaining libs. Also, I got a few test failures with this branch, but I don't know if they're at all related (shouldn't be...)
Since we are apparently already moving 9.0 towards a release candidate, this will probably have to wait a bit.
Updated to remove libpari.a. Apparently we had been explicitly installing it pretty much forever, so I couldn't find a clear record of why, but I don't think there's any need for it currently.
My only concern here is there no easy way (since it is not a flag that can be passed to PARI_CONFIGURE) to force installation of the static lib. But if someone asks for it for some reason it can be added later.
Replying to @embray:
Updated to remove libpari.a. Apparently we had been explicitly installing it pretty much forever, so I couldn't find a clear record of why, but I don't think there's any need for it currently.
pari/gp upstream claims that statically linked gp is faster than dynamically linked one.
Description changed:
---
+++
@@ -59,3 +59,5 @@
* ecm: apparently we deliberately build it as a static lib by default; not sure why. Need to investigate (performance reasons?)
* gap: The SmallGrp package also installs some ".a" file that are not static libs; otherwise GAP is not installing any static libs (some things in gap_packages might but I haven't checked yet).
+
+* symmetrica: Only built as a static library and linked statically into Sage's symmetrica module; it's not used anywhere else at the moment so this is fine for now. There has been discussion about taking over maintenance of symmetrica and updating its build system, so this can be looked into more as a separate issue.Replying to @dimpase:
Replying to @embray:
Updated to remove libpari.a. Apparently we had been explicitly installing it pretty much forever, so I couldn't find a clear record of why, but I don't think there's any need for it currently.
pari/gp upstream claims that statically linked gp is faster than dynamically linked one.
If so, it appears we are already not doing this, though we could. Regardless, this can be done during PARI/GP build: there is no need to install the static lib after the fact.
Description changed:
---
+++
@@ -60,4 +60,6 @@
* gap: The SmallGrp package also installs some ".a" file that are not static libs; otherwise GAP is not installing any static libs (some things in gap_packages might but I haven't checked yet).
+* python2/3: The static lib installation (performed by the `libainstall` make target) is a part of Python's standard installation process and more trouble to remove than it's worth, although I don't think we explicitly need it for anything.
+
* symmetrica: Only built as a static library and linked statically into Sage's symmetrica module; it's not used anywhere else at the moment so this is fine for now. There has been discussion about taking over maintenance of symmetrica and updating its build system, so this can be looked into more as a separate issue.Description changed:
---
+++
@@ -56,6 +56,8 @@
* cliquer: it already passes `--disable-static` but it installs a test file into `share/` which happens to end with `.a` but is not a static lib as far as I can tell.
+* ecl: The `.a` modules installed with ECL seem to be pretty baked into its build process and I'm not sure if there's an easy way to omit them or if it's even desirable to.
+
* ecm: apparently we deliberately build it as a static lib by default; not sure why. Need to investigate (performance reasons?)
* gap: The SmallGrp package also installs some ".a" file that are not static libs; otherwise GAP is not installing any static libs (some things in gap_packages might but I haven't checked yet).libbraiding and libhomfly will be handled by this as well once #28927 is in.
Description changed:
---
+++
@@ -65,3 +65,5 @@
* python2/3: The static lib installation (performed by the `libainstall` make target) is a part of Python's standard installation process and more trouble to remove than it's worth, although I don't think we explicitly need it for anything.
* symmetrica: Only built as a static library and linked statically into Sage's symmetrica module; it's not used anywhere else at the moment so this is fine for now. There has been discussion about taking over maintenance of symmetrica and updating its build system, so this can be looked into more as a separate issue.
+
+* yasm: If I understand correctly, libyasm contains some special support functions and is intended to be linked statically.Description changed:
---
+++
@@ -62,7 +62,13 @@
* gap: The SmallGrp package also installs some ".a" file that are not static libs; otherwise GAP is not installing any static libs (some things in gap_packages might but I haven't checked yet).
+* mpir: mpir/gmp is linked statically for at least some libraries such as ECM, and possibly some others (GP?).
+
+* ntl: NTL's build system makes static library installation pretty mandatory--I don't know if there's any good reason for that or not, so for now we can just leave it alone.
+
* python2/3: The static lib installation (performed by the `libainstall` make target) is a part of Python's standard installation process and more trouble to remove than it's worth, although I don't think we explicitly need it for anything.
+
+* ratpoints: Has only ever supported being built as a static library, and is statically linked into Sage's ratpoints interface.
* symmetrica: Only built as a static library and linked statically into Sage's symmetrica module; it's not used anywhere else at the moment so this is fine for now. There has been discussion about taking over maintenance of symmetrica and updating its build system, so this can be looked into more as a separate issue.
I think that with the last commit this pretty much covers everything. I looked into all the other packages that are installing static libs, and determined that it's either on purpose, or too much trouble to mess with as part of this ticket.
Reviewer: Marc Mezzarobba
Thank you again!
Changed branch from u/embray/ticket-28890 to f99562a
****************************************************
Configuring zlib-1.2.11.p0
unknown option: --disable-static
./configure --help for help
** ./configure aborting.
unknown option: --disable-static
./configure --help for help
** ./configure aborting.
********************************************************************************
Error configuring zlib-1.2.11.p0
********************************************************************************
Changed branch from f99562a to u/embray/ticket-28890
Hrm, that's annoying. Thanks, zlib...
Branch pushed to git repo; I updated commit sha1. New commits:
29445a8 | Trac #28890: Don't use sdh_configure to configure zlib; it does not use |
New commits:
29445a8 | Trac #28890: Don't use sdh_configure to configure zlib; it does not use |
I'm getting libpng build errors on KuCalc (Debian 9 x86_64) because something is wrong with zlib dynamic linking:
libtool: link: gcc -fPIC -g -Wl,-rpath -Wl,/var/lib/buildbot/slave/sage_git/build/local/lib -o .libs/pngunknown contrib/libtests/pngunknown.o -L/var/lib/buildbot/slave/sage_git/build/local/lib ./.libs/libpng16.so -lz -lm -Wl,-rpath -Wl,/var/lib/buildbot/slave/sage_git/build/local/lib
collect2: error: ld returned 1 exit status
Makefile:1026: recipe for target 'pngtest' failed
make[5]: *** [pngtest] Error 1
./.libs/libpng16.so: undefined reference to `inflateValidate'
I presume build/pkgs/libpng/spkg-install needs LDFLAGS set to point at $SAGE_LOCAL/lib/
in the case of zlib not coming from the system.
(and the same for the other zlib deps, I guess)
Replying to @dimpase:
I presume build/pkgs/libpng/spkg-install needs LDFLAGS set to point at $SAGE_LOCAL/lib/
in the case of zlib not coming from the system.
(and the same for the other zlib deps, I guess)
Don't we already do that for all packages? The failing command Volker posted already had -L/var/lib/buildbot/slave/sage_git/build/local/lib as well as -lz.
could it be the order of libs issue?
Or something to do with libpng using to carry its own zlib...
Batch modifying tickets that will likely not be ready for 9.1, based on a review of the ticket title, branch/review status, and last modification date.
Changed branch from u/embray/ticket-28890 to u/mkoeppe/ticket-28890
Description changed:
---
+++
@@ -70,6 +70,4 @@
* ratpoints: Has only ever supported being built as a static library, and is statically linked into Sage's ratpoints interface.
-* symmetrica: Only built as a static library and linked statically into Sage's symmetrica module; it's not used anywhere else at the moment so this is fine for now. There has been discussion about taking over maintenance of symmetrica and updating its build system, so this can be looked into more as a separate issue.
-
* yasm: If I understand correctly, libyasm contains some special support functions and is intended to be linked statically.I've removed the note about symmetrica from the description, since we took over maintenance and added an autotools build system.
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
Dusted off this ticket. It will be helpful to reduce the size of wheels in #31396
Changed reviewer from Marc Mezzarobba to Marc Mezzarobba, https://github.com/mkoeppe/sage/actions/runs/732049335
Tests running at https://github.com/mkoeppe/sage/actions/runs/732049335
A typo in the change to zlib broke lots of platforms
Branch pushed to git repo; I updated commit sha1. New commits:
9622d83 | build/pkgs/zlib/spkg-install.in: Fixup |
Branch pushed to git repo; I updated commit sha1. New commits:
3d701f8 | build/pkgs/zlib/spkg-install.in: Handle errors from configure |
Changed reviewer from Marc Mezzarobba, https://github.com/mkoeppe/sage/actions/runs/732049335 to Marc Mezzarobba, https://github.com/mkoeppe/sage/actions/runs/733869032
Now testing at https://github.com/mkoeppe/sage/actions/runs/733869032
Changed author from Erik Bray to Erik Bray, Matthias Koeppe
On OS X (with a bunch of homebrew packages), this prevents building of the following:
libbraiding.a
libcdd.a
libcddgmp.a
libec.a
libecm.a
libfplll.a
libgf2x.a
libgivaro.a
libhomfly.a
libiml.a
liblrcalc.a
libpari.a
libplanarity.a
libratpoints.a
librw.a
libsymmetrica.a
libecm.a and libratpoints.a are still built, as it says in the ticket description.
We should probably build without as many homebrew packages, to make sure things still work.
With ./configure --with-system-python3=no --with-system-ntl=no --with-system-gmp=no, I see libgmp.a, libgmpxx.a, and libntl.a still present. I think this is still consistent with the ticket description.
The runs at https://github.com/mkoeppe/sage/actions/runs/733869032 (linux/macos), https://github.com/mkoeppe/sage/actions/runs/732049334 (optional packages), were also successful.
On cygwin-standard (https://github.com/mkoeppe/sage/runs/2309214385), there is an issue with the standard package libbraiding - for which building shared libraries is broken (#30814)
Branch pushed to git repo; I updated commit sha1. New commits:
56178ab | build/bin/sage-dist-helpers (sdh_configure): On Cygwin, do not use --disable-static (until #30814 is done) |
New test on cygwin at https://github.com/mkoeppe/sage/actions/runs/738661838
Changed reviewer from Marc Mezzarobba, https://github.com/mkoeppe/sage/actions/runs/733869032 to Marc Mezzarobba, https://github.com/mkoeppe/sage/actions/runs/738661838
Branch pushed to git repo; I updated commit sha1. New commits:
c8ccab6 | build/bin/sage-dist-helpers: Fixup: = instead of == |
Changed reviewer from Marc Mezzarobba, https://github.com/mkoeppe/sage/actions/runs/738661838 to Marc Mezzarobba, John Palmieri
I'm not sure why I can't view the branch attached to this ticket, but it merges cleanly with Sage 9.3. I think we should add this to an early beta of 9.4 (or 9.5 if 9.4 is a bug-fix release) to make sure it gets tested thoroughly.
Thanks!
Changed branch from u/mkoeppe/ticket-28890 to c8ccab6