SSL certificates
Closed this issue · 19 comments
Great project!
My cabal.project refers to remote repositories using source-repository-package
, but cabal can't fetch them inside this container because the certificates aren't configured.
SSL certificate problem: unable to get local issuer certificate
Is it possible to add certificates to this container so that source-repository-package
can be used?
Hi, thanks!
That is actually a recent addition to this project, have you checked the version v8
(released a week or so ago)? In that one I added the support for SSL certificates, and this test case seemed to work:
ghc-musl/example/cabal.project
Lines 1 to 3 in e923d1e
If the v8
also doesn't work, would you be able to provide a repository or a cabal.project
snippet which reproduces the issue?
I'm using utdemir/ghc-musl:v8-libgmp-ghc8101
Is that the v8 you mean?
You can reproduce as follows:
git clone -b rc/test-ghc-musl https://github.com/jgm/pandoc tmp-pandoc
cd tmp-pandoc
make debpkg
Results also visible here:
https://github.com/jgm/pandoc/runs/915985898?check_suite_focus=true
Forgive me, I'm a docker ignoramus, so I may be doing things wrong.
That definitely looks like a bug on this side, and thanks for the test case. I'll take a look at it later today.
So, I spent some time looking at it. The SSL certificate issue was simple enough, turns out for some reason curl
doesn't see the certificates. I'll release a new version with the fix, but before that there is an easy workaround: you can set SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
environment variable which should fix the certificate issue.
However, even after this fix compiling one of the dependencies Diff
fails with a linker error:
/nix/store/8fcf4nm3v9jli60hk687z2cgz5dn1gds-binutils-2.31.1/bin/ld: /nix/store/3yxr8wfj 892xs2yxmf4hzbzlrwr5y79h-gcc-9.3.0/lib/gcc/x86_64-unknown-linux-musl/9.3.0/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/nix/store/8fcf4nm3v9jli60hk687z2cgz5dn1gds-binutils-2.31.1/bin/ld: /nix/store/3yxr8wfj 892xs2yxmf4hzbzlrwr5y79h-gcc-9.3.0/lib/gcc/x86_64-unknown-linux-musl/9.3.0/crtend.o: relocation R_X86_64_32 against `.ctors' can not be used when making a shared object; recompile with -fPIC
/nix/store/8fcf4nm3v9jli60hk687z2cgz5dn1gds-binutils-2.31.1/bin/ld: final link failed: nonrepresentable section on output
collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1)
I'll take a look at it later this week. Likely we'll need to pass some flags when building some/all upstream libraries.
Edit: I noticed that inside a fresh, empty container cabal new-install --enable-static --enable-executable-static pandoc
finishes successfully, so it seems to me that there is no major issue, we just need to figure out the correct cabal
incantation.
Great, thanks for looking into it further.
I just released v10
which fixes the SSL certificate issue. Now I'm looking at the linker error.
@jgm I got it compiling! The fix was using cabal's --enable-executable-static
flag instead of --ghc-options
:
diff --git a/linux/make_artifacts.sh b/linux/make_artifacts.sh
index 9dca400af..e248138ff 100644
--- a/linux/make_artifacts.sh
+++ b/linux/make_artifacts.sh
@@ -16,7 +16,7 @@ ghc --version
cabal v2-update
cabal v2-clean
-cabal v2-configure --enable-tests -f-export-dynamic -fstatic -fembed_data_files -fbibutils --ghc-options '-optc-Os -optl=-pthread -optl=-static -fPIC -split-sections' . pandoc-citeproc
+cabal v2-configure --enable-tests -f-export-dynamic -fstatic -fembed_data_files -fbibutils --enable-executable-static --ghc-options '-optc-Os -optl=-pthread -split-sections' . pandoc-citeproc
cabal v2-build . pandoc-citeproc
cabal v2-test -j1 . pandoc-citeproc
for f in $(find dist-newstyle -name 'pandoc*' -type f -perm +400); do cp $f /artifacts/; done
Also, a few notes:
- I had to bump the
pandoc-citeproc
revision since the one tagged incabal.project
did not compile, but you fixed the issue on later revisions. - The
commonmark-hs
override oncabal.project
is ignored because it has preceding whitespace. - After the build finishes;
make_artifacts.sh
script goes on trying to build a deb package, which requires a bunch of tools which doesn't exist inside the container. I'm going to go ahead and add a few which I think would be useful in general case (things likefind
andxargs
), but I am not planning to add things likefakeroot
ordpkg-deb
to the container. So, my solution would be to get that part of the script to run outside of the container. I hope this works for you.
For reference, here is all the changes I did to get it working: https://gist.github.com/cd6d360059137ed8baa40874f8ca4881
Let me know if there is anything else I can do to help you; I used pandoc
countless times and this is the least I can do. Thanks!
That's really great, thanks for tracking this down! I must say, it's very convenient to be able to create the deb in the same container, so without the ability to do that I may end up sticking with my current home-grown system based on alpine, though it's only ghc 8.6.5. I'll have to think about the tradeoffs of moving it outside the container.
@jgm Fair enough, makes sense. Let me know if you can not find an easier solution, I can put dpkg-deb
and fakeroot
to the container if there is no better way.
What would be the drawbacks of adding these to the container? (Or adding a way, such as apt
or apk
, to fetch them?) I notice that not even tar
seems to be present. With just a few more tools it would be possible to use this container not just to compile but to package in canonical formats for linux.
What would be the drawbacks of adding these to the container?
Nothing major, mainly the maintenance cost. For most tools like tar
, I have no concerns of adding them. However, more complicated tools sometimes require some work to get it run well inside the container. eg. when I'm try to add the dependencies you need, tar
and dpkg-tools
worked just fine; however fakeroot
did not work straightaway, now I need to figure out what's wrong, add some tests so they won't break again etc.
Or adding a way, such as apt or apk, to fetch them?
This might be a good option to. I could get nix
working inside the container, so it should make it possible to install new libraries. Maybe apk
would work too.
So, in short; there is no major issue other than finding spare time. I guess it would be wisest to spend time on making the container extensible so these things do not end up being a blocker. But, to be honest, this likely won't happen earlier than this weekend, sorry.
That would be great! There's absolutely no urgency on my end, since I already have a working build in a different container.
@jgm, I released v12
which contains tar
, deb
and fakeroot
. make debpkg
now succeeds on the branch you mentioned. However, I noticed that you removed the calls to fakeroot
and dpkg
, so I haven't tested them. Let me know how it goes.
Regarding to extensibility, apk
package manager did not work straightaway inside the container, so it requires some more effort.
That's great news. I'll test right now.
OK, I got a strange error:
dpkg-deb: building package 'pandoc' in '/artifacts/pandoc-2.10.1-1-amd64.deb'.
1990
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate memory
1991
dpkg-deb: error: <compress> from tar -cf subprocess returned error exit status 2
1992
make: *** [debpkg] Error 2
Any ideas?
Here's the full build log, though it's not too informative.
https://github.com/jgm/pandoc/runs/969126050?check_suite_focus=true
Interesting error. I googled a bit and they tell that higher compression levels might use a lot of memory (if this is indeed caused from a lack of memory). I don't know how much memory GH actions provide, but maybe adjusting these flags might help:
-zcompress-level
Specify which compression level to use on the compressor backend, when building a package (default is 9 for gzip, 6 for xz). The accepted values are 0-9
with: 0 being mapped to compressor none for gzip. Before dpkg 1.16.2 level 0 was equivalent to compressor none for all compressors.
-Scompress-strategy
Specify which compression strategy to use on the compressor backend, when building a package (since dpkg 1.16.2). Allowed values are none (since dpkg 1.16.4),
filtered, huffman, rle and fixed for gzip (since dpkg 1.17.0) and extreme for xz.
-Zcompress-type
Specify which compression type to use when building a package. Allowed values are gzip, xz (since dpkg 1.15.6), and none (default is xz).
First, I'd probably try something like -Zgzip -z4
.
Strange: compiling pandoc uses a huge amount of memory. It's hard for me to believe that a machine with enough memory to compile pandoc doesn't have enough memory to compress the binary.
But I'll try it with the flags you suggest!
It worked specifying gzip
instead of xz
(even with level 9 compression).
The resulting deb is quite a bit bigger than before, but I can live with that.
Thanks for all your help with this!
I'm glad it worked! And I'm sorry it took a bit long, let me know if there's anything else I can do to make things easier.