Unable to bundle install latest psych 5.0.0 on github actions
danielwestendorf opened this issue · 31 comments
Ensure the following before filing this issue
-
I verified it reproduces with the latest version with
- uses: ruby/setup-ruby@v1(see Versioning policy) -
I tried to reproduce the issue locally by following the workflow steps (including all commands done by
ruby/setup-ruby, except forDownloading Ruby&Extracting Ruby),
and it did not reproduce locally (if it does reproduce locally, it's not a ruby/setup-ruby issue)
Are you running on a GitHub-hosted runner or a self-hosted runner?
GitHub-hosted runner
The workflow code or a link to the workflow file
Generated via GH web UI Actions > Ruby > Configure and commenting out the test line.
https://github.com/danielwestendorf/gh-actions-ruby-psych-5/blob/main/.github/workflows/ruby.ymlLink to the log of a failed workflow job, or to a gist with the output
The command and output of the failing step
bundle install
/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle config --local path /home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle
/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle lock
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies...
Writing lockfile to /home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/Gemfile.lock
Cache key: setup-ruby-bundler-cache-v4-ubuntu-20.04-ruby-2.7.6-Gemfile.lock-32b0ae944f0f9b2d8f3c61f68f6d3cb975db47fe2ae2efbf5bf6fa7b2f6a4dbe
/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle install --jobs 4
Fetching gem metadata from https://rubygems.org/........
Using bundler 2.3.26
Fetching stringio 3.0.2
Installing stringio 3.0.2 with native extensions
Fetching psych 5.0.0
Installing psych 5.0.0 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory:
/home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle/ruby/2.7.0/gems/psych-5.0.0/ext/psych
/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/ruby -I
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0 -r
./siteconf20221205-1835-1muibuj.rb extconf.rb
checking for yaml.h... no
yaml.h not found
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/$(RUBY_BASE_NAME)
--with-libyaml-source-dir
--without-libyaml-source-dir
--with-yaml-0.1-config
--without-yaml-0.1-config
--with-pkg-config
--without-pkg-config
--with-libyaml-dir
--without-libyaml-dir
--with-libyaml-include
--without-libyaml-include=${libyaml-dir}/include
--with-libyaml-lib
--without-libyaml-lib=${libyaml-dir}/lib
To see why this extension failed to compile, please check the mkmf.log which can
be found here:
/home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/psych-5.0.0/mkmf.log
extconf failed, exit code 1
Gem files will remain installed in
/home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle/ruby/2.7.0/gems/psych-5.0.0
for inspection.
Results logged to
/home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/psych-5.0.0/gem_make.out
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:99:in
`run'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/ext_conf_builder.rb:48:in
`block in build'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/tempfile.rb:291:in `open'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/ext_conf_builder.rb:30:in
`build'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:169:in
`block in build_extension'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:165:in
`synchronize'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:165:in
`build_extension'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:210:in
`block in build_extensions'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:207:in
`each'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:207:in
`build_extensions'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/installer.rb:844:in
`build_extensions'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/source/rubygems.rb:207:in
`install'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/installer/gem_installer.rb:54:in
`install'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/installer/parallel_installer.rb:186:in
`do_install'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/installer/parallel_installer.rb:177:in
`block in worker_pool'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:62:in
`apply_func'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:57:in
`block in process_queue'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:54:in
`loop'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:54:in
`process_queue'
/opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:91:in
`block (2 levels) in create_threads'
An error occurred while installing psych (5.0.0), and Bundler cannot continue.
In Gemfile:
psych
Took 5.42 seconds
Error: Error: The process '/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle' failed with exit code 5
at ExecState._setResult (/home/runner/work/_actions/ruby/setup-ruby/0a29871fe2b0200a17a4497bae54fe5df0d973aa/dist/index.js:6023:25)
at ExecState.CheckComplete (/home/runner/work/_actions/ruby/setup-ruby/0a29871fe2b0200a17a4497bae54fe5df0d973aa/dist/index.js:6006:18)
at ChildProcess.<anonymous> (/home/runner/work/_actions/ruby/setup-ruby/0a29871fe2b0200a17a4497bae54fe5df0d973aa/dist/index.js:5900:27)
at ChildProcess.emit (node:events:390:28)
at maybeClose (node:internal/child_process:1064:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)Any other notes?
No response
A good place to check are the Actions workflows for the repo, in this case, ruby/psych. It looks like libyaml-dev is not installed?
In general, when building Ruby or any extension gems (like psych), there are often build dependencies. They may not be included with the 'base' Actions image.
As it says:
checking for yaml.h... no
yaml.h not found
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
So you'll need to install libyaml-dev or so, or use --libyaml-source-dir.
cc @hsbt --libyaml-source-dir should be documented in https://github.com/ruby/psych/blob/master/README.md probably. But even then it's not really convenient to get libyaml.
Shouldn't installing Pysch 5.0.0 on Ruby 2.7.6 just work? Thanks to ruby/psych#550 or am I missing something?
Psych version 4 has yaml bundled. Psych version 5 does not?
See https://github.com/ruby/psych/tree/4-0-stable/ext/psych/yaml vs https://github.com/ruby/psych/tree/v5.0.0/ext/psych/yaml
Oh I thought Ruby itself bundled libyaml
3.1 does, master does not
but Pysch 5.0.0 can't use it?
I suppose it could, never tried. Vendoring c source for dependencies has pros and cons, and other default extension gems do not vendor, like Ruby's OpenSSL.
Oh I thought Ruby itself bundled libyaml
It was just psych itself vendoring libyaml.
So this is an expected breaking change for Psych >= 5, that it needs a system libyaml-dev installed (like openssl indeed).
Can we install libyaml-dev into https://github.com/actions/runner-images by ruby/setup-ruby layer?
I'm not sure how handle the additional packages like libffi, libgmp and libyaml in ruby/setup-ruby.
Yes. Similar code is in setup-ruby-pkgs, but the packages are user defined. Windows builds already have all packages installed by setup-ruby. By packages, I mean Ruby dependencies, but you knew that.
Briefly looked at ruby/ruby CI, libncurses5-dev & libyaml-dev are needed with Ubuntu, but not macOS?
Can we install
libyaml-devinto https://github.com/actions/runner-images byruby/setup-rubylayer?
I don't think setup-ruby should install any system package, for various reasons including time (it's a non-trivial overhead and in many cases not needed) and confusing side effects of setup-ruby (i.e. some extra system packages installed after the action).
If Ruby would need a new system package to even run then IMHO we should ask to preinstall that package in https://github.com/actions/runner-images rather than installing that package on every usage of setup-ruby.
Just like one needs to install a system package when using mysql2 or pg gem, now one needs to sudo apt-get install libyaml-dev before installing psych 5+.
I want to strongly encourage this group to please push this new library requirement down the stack (to runner-images if that's the right place) and not up the stack to the individuals trying to maintain CI.
Asking users of psych to add a step to every pipeline (or alternatively to use setup-ruby-pkgs) is a considerable cost in terms of human time, and a collectively large amount of clock time and cpu time to install it. Especially for indirect users of psych (e.g., a project requires rdoc which then transitively requires psych).
I'm not complaining, but as a data point from someone who is familiar with the problem and the workarounds, I've already spent a few hours trying to get just one project's pipelines green again.
Is it common to have an unbounded dependency on psych though? Because otherwise it's not a problem.
Regarding rdoc I think we should fix https://github.com/ruby/rdoc/blob/c5e7ee03ac3035b001d58e12564c1ea79540c3a8/rdoc.gemspec#L232, I think either it should not specify the dependency and use the default gem, or it should pin to a specific major version (e.g., 4) to avoid such incompatible changes, cc @aycabta for ruby/rdoc@ebe185c.
I think it's worth a try to create an issue at https://github.com/actions/runner-images to ask to preinstall libyaml-dev. It's probably very small in size and I believe the libyaml package is already included. I would like someone else to create that issue though (I don't personally have the need for it).
See 'Ubuntu : Update the libyaml version to 0.2.5 (latest)' in actions/runner-images
Because of this change, the logger CI is now failing.
https://github.com/ruby/logger
This probably won't be the only case.
We fixed it by removing rdoc from the development dependency list.
I'll be devil's advocate here, undecided whether setup-ruby should handle this. This type of issue may happen more in the future.
-
What should be installed, 0.2.2 (standard) or 0.2.5 (available from kinetic)?
-
I suspect that caching the packages in a GitHub release might be a good idea, not sure. If they are cached, more code somewhere.
Off-topic: Puma has a bit of 'app reload' logic, and upgrading bundled extension gems in the main process causes issues. Our yaml needs were trivial, so we replaced Psych with pure Ruby code. Maybe rdoc can do the same?
FWIW I measured how long sudo apt install libyaml-dev takes and it's 5-6 seconds.
That's way too much to add by default for setup-ruby.
Also from @MSP-Greg's last reply it comforts me in my position that we certainly don't want additional complexity and risks here (using a package from another Ubuntu version feels very risky, especially if the existing /usr/lib/libyaml.so is of a different version).
My reasoning is if people can install system packages of sqlite3/mysql2/pg/etc, they can for psych too.
And that has other advantages including more flexibility and e.g. caching via this action I noticed today.
I understand it's annoying and possibly more frequent but this is not the place to solve it, this action sets up Ruby it doesn't guess system dependencies or do anything about them.
As I said in #409 (comment), I think the only way to make it nicer is to ask to add libyaml-dev in https://github.com/actions/runner-images.
Oh and it looks like I was very lucky with thet 5-6 second run, because in 3 other runs it's more like 15 seconds:
https://github.com/eregon/setup-ruby/actions/runs/3649371803/jobs/6163957728
https://github.com/eregon/setup-ruby/actions/runs/3649348737/jobs/6163903717
https://github.com/ruby/rdoc/actions/runs/3648958399/jobs/6163011270
(FWIW that's a big part of why I don't like apt, dnf is so much faster/better)
Seems an additional reason to add it to runner images.
using a package from another Ubuntu version feels very risky, especially if the existing /usr/lib/libyaml.so is of a different version
Just to clarify, if we used kinetic, we'd need to install both libyaml & libyaml-dev...
I suspect runner-images may not want to add libyaml-dev, not sure. I'll see...
Help - can anyone verify whether libyaml & libyaml-dev are a normal part of a macOS install? Looking at the macOS workflow in ruby/ruby, can't see any mention of it? Maybe I need more coffee...
Or, one can use apt list --installed on Ubuntu. What's the equivalent on macOS?
Actually runner images provide some versions of Ruby, so that guarantees libyaml (i.e., the .so but not the headers) will be available on runner images, and the same for all required dependencies of ruby (or if one is missing it's an obvious issue and I'd think that would be quickly fixed in runner-images).
@MSP-Greg brew list (see https://apple.stackexchange.com/questions/101090/list-of-all-packages-installed-using-homebrew)
The macOS system might or not have libyaml by default, I have no idea.
On my dev machine:
$ brew list libyaml -v
/opt/homebrew/Cellar/libyaml/0.2.5/INSTALL_RECEIPT.json
/opt/homebrew/Cellar/libyaml/0.2.5/License
/opt/homebrew/Cellar/libyaml/0.2.5/.brew/libyaml.rb
/opt/homebrew/Cellar/libyaml/0.2.5/Changes
/opt/homebrew/Cellar/libyaml/0.2.5/include/yaml.h
/opt/homebrew/Cellar/libyaml/0.2.5/ReadMe.md
/opt/homebrew/Cellar/libyaml/0.2.5/lib/pkgconfig/yaml-0.1.pc
/opt/homebrew/Cellar/libyaml/0.2.5/lib/libyaml.dylib
/opt/homebrew/Cellar/libyaml/0.2.5/lib/libyaml.a
/opt/homebrew/Cellar/libyaml/0.2.5/lib/libyaml-0.2.dylib
So this is the equivalent of both libyaml and libyaml-dev
I suspected as much, thanks for verifying.
I don't see a libyaml installation on the core macOS system (but I may not be looking in all the places):
$ find /usr /System/Library /System/Developer -name 'yaml.h' 2> /dev/null
(no output)
$ find /usr /System/Library /System/Developer -name 'libyaml*' 2> /dev/null
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/share/ri/2.6.0/system/Psych/libyaml_version-c.ri
so I'm guessing it's a brew-installed package.
Interesting. You mentioned some CI that failed. Did it fail on Ubuntu and pass on macOS? I assume GitHub Actions?
I opened an issue, actions/runner-images#6725, and I hope 'parity' between Ubuntu & macOS is a valid reason...
@MSP-Greg Correct, the failures have only been on Ubuntu images.
Closing the loop here: libyaml-dev should be on runner images sometime this week: