galaaz-ggplot: issue with building and running demo

Question

galaaz-ggplot: issue with building and running demo

neomatrix369 opened this issue 3 years ago · 35 comments

On the back of the PR #93, after installing TruffleRuby via the gu tool and following the steps mentioned on https://github.com/graalvm/graalvm-demos/tree/master/galaaz-ggplot#introduction, the step to install the rspec gem initially failed due to the absence (or not being able to find these files:

libssl.so.10
libcrypto.so.10

Later when fixed via commit 714e155 and rerunning gem install rspec, worked fine.

The next step of installing the galaaz gem fails:

root@47c89260ca9a:/graalvm-demos/galaaz-ggplot# gem install galaaz
ERROR:  Error installing galaaz:
	invalid gem: package is corrupt, exception while verifying: invalid byte sequence in US-ASCII (ArgumentError) in /graalvm/languages/ruby/lib/gems/cache/galaaz-0.5.0.gem

Would need further investigation.

Answer 1 · 2021-10-12T15:53:20.000Z

714e155 is a huge hack, please do not do that.
Instead, run the post-install hook as documented and as shown in the post-install message:
https://github.com/oracle/truffleruby/blob/master/doc/user/installing-graalvm.md#installing-ruby-and-other-languages

Answer 2 · 2021-10-12T15:57:44.000Z

I can gem install galaaz fine on the latest snapshot:

$ gem i galaaz
Fetching galaaz-0.5.0.gem
Successfully installed galaaz-0.5.0
1 gem installed

Make sure the locale is set to something sensible, i.e., LANG=en_US.UTF-8.
Otherwise it fails and it warns:

$ LANG=C gem i galaaz
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[ruby] WARNING: Encoding.find('locale') is US-ASCII, this often indicates that the system locale is not set properly. Set LANG=C and LC_ALL=C to suppress this warning (but some things might break).
ERROR:  Error installing galaaz:
	invalid gem: package is corrupt, exception while verifying: invalid byte sequence in US-ASCII (ArgumentError) in /home/eregon/.rubies/truffleruby-dev/lib/gems/cache/galaaz-0.5.0.gem

See https://github.com/oracle/truffleruby#dependencies and https://github.com/oracle/truffleruby/blob/d16a373a1749ab28ad08989ef6351c0c6afa9055/tool/docker-configs.yaml#L17-L22 for Debian images

Answer 3 · 2021-10-12T16:07:52.000Z

714e155 is a huge hack, please do not do that. Instead, run the post-install hook as documented and as shown in the post-install message: https://github.com/oracle/truffleruby/blob/master/doc/user/installing-graalvm.md#installing-ruby-and-other-languages

Thanks for the response Beniot.

I totally agree, with it, I'm not a fan of it either, it should detect and recognise those libraries automatically but it does not. I have run the post_install_hook.sh script (this one https://github.com/oracle/truffleruby/blob/master/lib/truffle/post_install_hook.sh) following the docs, but it fails, I will follow your steps, and try again and report back.

Answer 4 · 2021-10-12T16:16:05.000Z

it should detect and recognise those libraries automatically but it does not.

It can't and it shouldn't do that. The point is these libssl/libcrypto have different ABI versions and it's unsafe to use them without recompiling the openssl C extension, which is what the post_install_hook.sh does.

Answer 5 · 2021-10-12T16:30:16.000Z

it should detect and recognise those libraries automatically but it does not.

It can't and it shouldn't do that. The point is these libssl/libcrypto have different ABI versions and it's unsafe to use them without recompiling the openssl C extension, which is what the post_install_hook.sh does.

Thanks I will change this as per your suggestions, I'm locating the linux dependencies/libraries now in order to support the post hook script

Answer 6 · 2021-10-12T16:38:14.000Z

it should detect and recognise those libraries automatically but it does not.

It can't and it shouldn't do that. The point is these libssl/libcrypto have different ABI versions and it's unsafe to use them without recompiling the openssl C extension, which is what the post_install_hook.sh does.

Success with running $GRAALVM_DIR/jre/languages/ruby/lib/truffle/post_install_hook.sh:

<--snipped-->
compiling ossl_x509name.c
compiling ossl_x509req.c
compiling ossl_x509revoked.c
compiling ossl_x509store.c
linking shared-object openssl.so
TruffleRuby was successfully installed in /graalvm/jre/languages/ruby

After installing dependencies build-essential, libssl1.0-dev and libcrypto*-dev - this looks right?

If these are correct, it would help to mention these dependencies on this page

Answer 7 · 2021-10-12T16:45:56.000Z

Yes, this looks good.
AFAIK only sudo apt-get install libssl-dev is needed as mentioned on https://github.com/oracle/truffleruby/blob/master/doc/user/installing-libssl.md

If these are correct, it would help to mention these dependencies on this page

They are already mentioned:
https://github.com/oracle/truffleruby/blob/master/doc/user/installing-graalvm.md#dependencies
But unfortunately that link is broken, I'll fix it. It should bring you to https://github.com/oracle/truffleruby/blob/master/README.md#dependencies

Answer 8 · 2021-10-12T16:46:33.000Z

BTW $GRAALVM_DIR/jre means you're using Java 8, but that's already unsupported for the 21.3 release, so I would suggest to use 11 instead.

Answer 9 · 2021-10-12T17:00:27.000Z

BTW $GRAALVM_DIR/jre means you're using Java 8, but that's already unsupported for the 21.3 release, so I would suggest to use 11 instead.
Yes the build I'm currently testing is Java 8 based, I will also run for Java 11. Maintaining Java 8 for demo apps that only run on Java 8 for now

Answer 10 · 2021-10-12T17:06:13.000Z

Yes, this looks good.

Thanks

AFAIK only sudo apt-get install libssl-dev is needed as mentioned on https://github.com/oracle/truffleruby/blob/master/doc/user/installing-libssl.md

If these are correct, it would help to mention these dependencies on this page

They are already mentioned: https://github.com/oracle/truffleruby/blob/master/doc/user/installing-graalvm.md#dependencies But unfortunately that link is broken, I'll fix it. It should bring you to https://github.com/oracle/truffleruby/blob/master/README.md#dependencies

apt-get install libssl-dev ==> might be apt-get install libssl1.0-dev (the latter worked for me on my distro, the former might be for older distros), is libcrypto-*dev also needed?

https://github.com/oracle/truffleruby/blob/master/README.md#dependencies

This link does not have all of the ones mentioned, I just captured in a fresh environment, I can do a PR to add the ones I found if you like

I noticed if we don't have the exact names, builds don't work as desired.

Answer 11 · 2021-10-12T17:20:43.000Z

apt-get install libssl-dev ==> might be apt-get install libssl1.0-dev (the latter worked for me on my distro, the former might be for older distros), is libcrypto-*dev also needed?

Did you try apt-get install libssl-dev?
We test that on Ubuntu 16.04, 18.04, 20.04 and debian:buster so I'm confident it works and it's enough.
libcrypto is just dependencies of libssl-dev AFAIK.

This link does not have all of the ones mentioned, I just captured in a fresh environment, I can do a PR to add the ones I found if you like

If there are missing ones it'd be useful to report it, e.g. as an issue and we'll add it to the Docker testing of TruffleRuby.

I noticed if we don't have the exact names, builds don't work as desired.

Do you mean for package names? What's the issue?

Answer 12 · 2021-10-12T17:22:51.000Z

https://packages.debian.org/buster/libssl-dev says

It contains development libraries, header files, and manpages for libssl and libcrypto.

Answer 13 · 2021-10-13T07:06:02.000Z

I've tested galaaz demo on macOS with graalvm-ee-java11-21.2.0.1. I Installed the Ruby component (gu install ruby), ran the post-install script (post_install_hook.sh), and finally added TruffleRuby to chruby with the chruby system command (as I'm using a Ruby manager, see https://github.com/oracle/truffleruby/blob/master/doc/user/ruby-managers.md#chruby).
Then both galaaz and rspec gems installed successfully.

The only annoying thing is the installation of all required dependencies ( ‘digest’, ‘base64enc’, ‘glue’, ‘stringi’, ‘mime’, etc.) , it takes so much time so I just terminated.

Answer 14 · 2021-10-13T10:29:37.000Z

I've tested galaaz demo on macOS with graalvm-ee-java11-21.2.0.1. I Installed the Ruby component (gu install ruby), ran the post-install script (post_install_hook.sh), and finally added TruffleRuby to chruby with the chruby system command (as I'm using a Ruby manager, see https://github.com/oracle/truffleruby/blob/master/doc/user/ruby-managers.md#chruby). Then both galaaz and rspec gems installed successfully.

The only annoying thing is the installation of all required dependencies ( ‘digest’, ‘base64enc’, ‘glue’, ‘stringi’, ‘mime’, etc.) , it takes so much time so I just terminated.

Did you rebuild the polyglot and libpolyglot images using gu rebuild-images polyglot libpolyglot (this is part of the post hook install step, see latter part of see https://github.com/oracle/truffleruby/blob/master/doc/user/installing-graalvm.md#dependencies) -- this takes a load of time on Linux (and inside the container, even with lots of RAM), at the moment it's failing to complete for me, I'll post the errors later on.

Answer 15 · 2021-10-13T10:30:48.000Z

apt-get install libssl-dev ==> might be apt-get install libssl1.0-dev (the latter worked for me on my distro, the former might be for older distros), is libcrypto-*dev also needed?

Did you try apt-get install libssl-dev? We test that on Ubuntu 16.04, 18.04, 20.04 and debian:buster so I'm confident it works and it's enough. libcrypto is just dependencies of libssl-dev AFAIK.

This link does not have all of the ones mentioned, I just captured in a fresh environment, I can do a PR to add the ones I found if you like

If there are missing ones it'd be useful to report it, e.g. as an issue and we'll add it to the Docker testing of TruffleRuby.

I noticed if we don't have the exact names, builds don't work as desired.

Do you mean for package names? What's the issue?

I found dependencies that I'm adding to the Dockerfile I will share separate once I have a successful build

Answer 16 · 2021-10-13T11:49:30.000Z

Did you rebuild the polyglot and libpolyglot images using gu rebuild-images polyglot libpolyglot (this is part of the post hook install step, see latter part of see https://github.com/oracle/truffleruby/blob/master/doc/user/installing-graalvm.md#dependencies) -- this takes a load of time on Linux (and inside the container, even with lots of RAM), at the moment it's failing to complete for me, I'll post the errors later on.

This is only necessary if you want to use polyglot/libpolyglot in --native mode.
Typically people use --jvm --polyglot as that removes the need to recompile any images.

Answer 17 · 2021-10-13T12:01:46.000Z

Did you rebuild the polyglot and libpolyglot images using gu rebuild-images polyglot libpolyglot (this is part of the post hook install step, see latter part of see https://github.com/oracle/truffleruby/blob/master/doc/user/installing-graalvm.md#dependencies) -- this takes a load of time on Linux (and inside the container, even with lots of RAM), at the moment it's failing to complete for me, I'll post the errors later on.

This is only necessary if you want to use polyglot/libpolyglot in --native mode. Typically people use --jvm --polyglot as that removes the need to recompile any images.

It would help to post this hint somewhere, as it was part of the instructions I went ahead trying to get it built.

Answer 18 · 2021-10-13T15:58:35.000Z

It would help to post this hint somewhere, as it was part of the instructions I went ahead trying to get it built.

OK, I tried to rework but noticed it's really not a good place to explain that and just remove that part.
I think rebuilding polyglot/libpolyglot is very rarely needed.

Answer 19 · 2021-10-13T16:06:51.000Z

This is only necessary if you want to use polyglot/libpolyglot in --native mode. Typically people use --jvm --polyglot as that removes the need to recompile any images.

It would help to post this hint somewhere, as it was part of the instructions I went ahead trying to get it built.

Would you like to see the failure messages when trying to rebuild the images, its related to blacklisted methods - it would be good to iron out this issue (separate from the demos stuff I'm looking at)

Answer 20 · 2021-10-13T16:10:04.000Z

Would you like to see the failure messages when trying to rebuild the images, its related to blacklisted methods - it would be good to iron out this issue (separate from the demos stuff I'm looking at)

Yes, please create a separate issue for that on oracle/graal.

Answer 21 · 2021-10-13T16:16:57.000Z

I found dependencies that I'm adding to the Dockerfile I will share separate once I have a successful build

I followed the steps mentioned in the docs kindly provided by you so far (above) and this is the outcome after running the post hook script:

root@f755aa023802:/graalvm-demos/galaaz-ggplot# gem install rspec
ERROR:  Loading command: install (RuntimeError)
	you may need to install the system OpenSSL library libssl - see https://github.com/oracle/truffleruby/blob/master/doc/user/installing-libssl.md (libssl.so.10: cannot open shared object file: No such file or directory)
ERROR:  While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args' for nil:NilClass

I tried to look for the installed libraries and you can see the version numbering is slightly different, GraalVM is looking for libssl.so.10 but the below are available instead:

root@f755aa023802:/graalvm-demos/galaaz-ggplot# find / -name *libssl.so*
/usr/lib/x86_64-linux-gnu/libssl.so.1.1
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.2

I must be missing a step I think or two

Answer 22 · 2021-10-13T17:14:34.000Z

I found dependencies that I'm adding to the Dockerfile I will share separate once I have a successful build

@eregon see this https://github.com/graalvm/graalvm-demos/pull/93/files#diff-68d59feb363a91e2eac51920698d7f8150af1ab0fdbdf4b5af51bccf821ba594R8, if any one of them stands out for your docs - thnx

Answer 23 · 2021-10-14T16:22:54.000Z

Are you sure you ran the post-install script? What was the output?
This looks like it didn't run or it failed.

Answer 24 · 2021-10-14T16:25:46.000Z

Having libssl 10 installed is not great and should be avoided, so there is less risk to mix headers files and libs of different versions (but it should still work).
Does the base image come with libssl.so.1.0.2? Which base image is it?

Answer 25 · 2021-10-14T17:40:50.000Z

Are you sure you ran the post-install script? What was the output? This looks like it didn't run or it failed.

<--snipped-->
compiling ossl_x509name.c
compiling ossl_x509req.c
compiling ossl_x509revoked.c
compiling ossl_x509store.c
linking shared-object openssl.so
TruffleRuby was successfully installed in /graalvm/jre/languages/ruby

I'll re-run and share the output but it ran successfully at that time. Here is the output of running post install hook, see
ruby_post_install_hook_sh.log @eregon

Is there some command we can run to check if this script did all the things it should have? I guess the gem install isn't really a test to check this is it?

Answer 26 · 2021-10-14T17:42:34.000Z

Having libssl 10 installed is not great and should be avoided, so there is less risk to mix headers files and libs of different versions (but it should still work). Does the base image come with libssl.so.1.0.2? Which base image is it?

The base image is from the docker image buildpack-deps:stretch-scm, when updated with apt-get it brings down the versions of the libraries you see above. I guess I should use debian:buster instead, could resolve these version issues.

Answer 27 · 2021-10-14T19:48:49.000Z

@eregon It looks better now, I think the debian:buster migration did it! Let me test it further to see if the app runs. Just now Im able to install my first gem without any errors.

Answer 28 · 2021-10-15T00:10:36.000Z

Both the gems (rspec and galaaz) install successfully after applying these changes: 2e1470e
f03fbea 0bcf891 5ac240e 8b7e9d5, in summary moving to debian:buster and installing the locale settings resolved the above reported issues.

But it still appears that if the underlying Linux distro is changed to another distro and the above steps may fail. Also setting locale, isn't a trivial step of setting LANG="xxxx" but adding the locale into the locale config and another step thereafter -- I suggest mentioning both these in the docs, otherwise it can be a struggle. I have had locale issues with other systems in the past, and it's fiddley and can take time to get it right.

Also, the demo needs a Linux dependency bzip2 in order for the galaaz ... command to progress, although now we have a Truffle/static lib related error when I run galaaz master_list:scatter_plot, see logs:

<---runs for 20 odd minutes, then the below error occurs -->

<--snipped-->
The downloaded source packages are in
	‘/tmp/Rtmpfj1YTD/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
unknown.r:1:in `library.dynam': libc++.so.1: cannot open shared object file: 
No such file or directory (com.oracle.truffle.nfi.backend.libffi.NFIUnsatisfiedLinkError) 
(RuntimeError)
Translated to internal error
	from unknown.r:1:in `loadNamespace'
<--snipped-->

Detailed logs, see galaaz_master_list_demo_error.log

Answer 29 · 2021-10-15T15:50:44.000Z

@eregon It looks better now, I think the debian:buster migration did it! Let me test it further to see if the app runs. Just now Im able to install my first gem without any errors.

Yeah, stretch is quite old, buster seems better and will avoid using the EOL'd libssl 1.0.

Answer 30 · 2021-10-15T15:54:42.000Z

I suggest mentioning both these in the docs, otherwise it can be a struggle.

I updated https://github.com/oracle/truffleruby/blob/master/doc/user/utf8-locale.md recently, I think that should be clear.

Could you open a separate issue for the new issue you get now?
Also that looks like an issue from R, not Ruby.

Answer 31 · 2021-10-18T22:57:46.000Z

I suggest mentioning both these in the docs, otherwise it can be a struggle.

I updated https://github.com/oracle/truffleruby/blob/master/doc/user/utf8-locale.md recently, I think that should be clear.

Could you open a separate issue for the new issue you get now? Also that looks like an issue from R, not Ruby.

Sorry, I didn't get back to you earlier, I have also been testing the changes locally.

I'll raise this new issue as an R related one, who is the R expert that I can talk to and liaise about this issue?

Answer 32 · 2021-10-19T11:14:46.000Z

To fix this error:

unknown.r:1:in `library.dynam': libc++.so.1: cannot open shared object file: No such file or directory (com.oracle.truffle.nfi.backend.libffi.NFIUnsatisfiedLinkError) (RuntimeError)

you will need to install the libc++-dev package on the system.

Answer 33 · 2021-10-19T11:32:35.000Z

To fix this error:

unknown.r:1:in `library.dynam': libc++.so.1: cannot open shared object file: No such file or directory (com.oracle.truffle.nfi.backend.libffi.NFIUnsatisfiedLinkError) (RuntimeError)

you will need to install the libc++-dev package on the system.

Thanks @steve-s, I'm rebuilding and testing it now

Answer 34 · 2021-10-19T17:17:28.000Z

To fix this error:
unknown.r:1:in `library.dynam': libc++.so.1: cannot open shared object file: No such file or directory (com.oracle.truffle.nfi.backend.libffi.NFIUnsatisfiedLinkError) (RuntimeError)
you will need to install the libc++-dev package on the system.
Thanks @steve-s, I'm rebuilding and testing it now

It's still building and installing the R components, it's taking some time but the above error hasn't occurred anymore, the fix seems to have fixed it.

Answer 35 · 2021-10-19T18:53:59.000Z

Build is complete ~45-60 minutes, scatter plot demo runs fine!