Ruby can't compile with psych when libyaml isn't installed

[eileencodes]

Hello! Since I pulled Ruby master yesterday I'm unable to configure or run make successfully due to an error that the digest file is missing. This only happens when libyaml isn't installed - if I install libyaml then everything works correctly. However the code that's failing is supposed to download libyaml and instead raises the following error:

/Users/eileencodes/open_source/ruby/tool/extlibs.rb:6:in `require': cannot load such file -- digest (LoadError)
	from /Users/eileencodes/open_source/ruby/tool/extlibs.rb:6:in `<top (required)>'
	from /Users/eileencodes/open_source/ruby/ext/psych/extconf.rb:22:in `require_relative'
	from /Users/eileencodes/open_source/ruby/ext/psych/extconf.rb:22:in `<top (required)>'
	from ./ext/extmk.rb:214:in `load'
	from ./ext/extmk.rb:214:in `block in extmake'
	from /Users/eileencodes/open_source/ruby/lib/mkmf.rb:324:in `open'
	from ./ext/extmk.rb:210:in `extmake'
	from ./ext/extmk.rb:572:in `block in <main>'
	from ./ext/extmk.rb:568:in `each'
	from ./ext/extmk.rb:568:in `<main>'
make[1]: *** [ext/psych/exts.mk] Error 1
make[1]: *** Waiting for unfinished jobs....

I know that there were a lot of recent changes to psych and libyaml is no longer bundled with it. On my machine where libyaml is not installed when I run make on my Ruby checkout (or configure) it will hit this line https://github.com/ruby/ruby/blob/master/ext/psych/extconf.rb#L22 which loads tool/extlibs.rb. That file then has a require 'digest' which is where it is failing to build and cannot load the file. Since it fails to require the digest file, the download_cache is never used. I can consistently reproduce if I don't have libyaml installed. If I manually install it with homebrew, Ruby builds fine.

Machine details:

1. MacOS Monterey, M1 Max
2. Ruby master as of github.com/ruby/ruby/0cb43034b416aaf28bf57ee34bc095fdcb659f33

Steps to reproduce:

1. Uninstall libyaml or ensure it is not installed
2. make clean
3. Run configure script, mine below:

autoconf
./configure --prefix=$HOME/.rbenv/versions/ruby-trunk --disable-install-rdoc --with-openssl-dir=$(brew --prefix openssl@1.1) optflags='-O3 -DNDEBUG' debug='-ggdb'
make -j16

4. You can also see it on each rebuild with make -j16 if it failed to configure so you don't need to re-run configure over and over.

[eileencodes]

When I run rake compile in psych with_config("libyaml-source-dir") returns nil and enable_config("bundled-libyaml", false) returns false so my machine falls through to configuring yaml from source here https://github.com/ruby/psych/blob/master/ext/psych/extconf.rb#L33-L55.

So when building psych inside psych it works fine, but when building psych inside Ruby it fails. Inside Ruby, with_config("libyaml-source-dir") returns true.

[nobu]

Doesn’t make extract-extlubs after configure make ext/psych/yaml-2.5?
Then ext/psych/extconf.rb should fall back to it.
Note that downloading in extconf.rb is for the separate repositories.

[eileencodes]

Doesn’t make extract-extlubs after configure make ext/psych/yaml-2.5?

I think you meant extlibs but I'm sure what's not working right.

When tool/extlibs.rb gets loaded there's a require 'digest' at the top, that's what's causing the download to not happen on my machine.

Note that downloading in extconf.rb is for the separate repositories

I'm not downloading it or running it manually, it's make that's running it.

[XrXr]

I get the same error on FreeBSD 13 when libyaml is not installed. make configure-ext fails, which make runs. Looks like it's because make configure-ext uses miniruby which doesn't have digest.

Running make extract-extlibs first let's me get past make configure-ext into make build-ext, but that fails due to libyaml having a Makefile BSD make dislikes:

--- ext/psych/all ---
--- libyaml.a ---
cd libyaml && make
make[3]: "/usr/home/ec2-user/ruby/ext/psych/libyaml/Makefile" line 884: Need an operator
make[3]: "/usr/home/ec2-user/ruby/ext/psych/libyaml/Makefile" line 888: Need an operator
make[3]: "/usr/home/ec2-user/ruby/ext/psych/libyaml/Makefile" line 890: Need an operator
make[3]: Fatal errors encountered -- cannot continue
make[3]: stopped in /usr/home/ec2-user/ruby/ext/psych/libyaml
*** [libyaml.a] Error code 1

[nobu]

@eileencodes

Doesn’t make extract-extlubs after configure make ext/psych/yaml-2.5?

I think you meant extlibs but I'm sure what's not working right.

Sorry, yes, it is.

When tool/extlibs.rb gets loaded there's a require 'digest' at the top, that's what's causing the download to not happen on my machine.

make extract-extlibs runs BASERUBY, which should have digest.

$ ruby -w tool/extlibs.rb ext
downloading for ext/fiddle/extlibs
.downloaded-cache/libffi-3.2.1.tar.gz already exists
downloading for ext/psych/extlibs
.downloaded-cache/yaml-0.2.5.tar.gz already exists

Note that downloading in extconf.rb is for the separate repositories

I'm not downloading it or running it manually, it's make that's running it.

I mean we are going to remove all extlibs files.
You will then need to install libffi and libyaml before building ruby.

[nobu]

@XrXr
Those lines seem coming from the original libyaml source, Makefile.in.
Could your report it there?

[c4lliope]

I see this error on Fedora 35, and during make extract-extlibs I had an error concerning the patch library.
sudo dnf install patch libyaml libffi seems to help.

[hsbt]

We completely disabled auto-download feature on ruby/psych and ruby/ruby. I think we will not see this issue in the future.