postmodern/chruby

A way to not set GEM_HOME if the installed ruby is under $HOME

eregon opened this issue · 21 comments

eregon commented

I would like a way to tell chruby to not set $GEM_* variables, as this is much safer (i.e., does not mix gems of different Ruby implementations) when e.g., developing TruffleRuby and executing two different Rubies without a chruby in between (see #410 (comment)).

Currently I'm doing this by using a branch on my chruby fork, removing the code that sets GEM_HOME, but that's obviously not very convenient or maintainable, and I can't easily advise other TruffleRuby developers to do the same. I'd much rather this was possible in chruby itself.

Actually, there is already a way to do this, by running everything as root, but that's obviously not very safe for my use-case.
So I'm thinking to just extend the check if (( UID != 0 )); then to something like if (( UID != 0 )) && [ "$CHRUBY_SET_GEM_HOME" != "false" ]; then.

We could also automatically just detect if $RUBY_ROOT is under $HOME and not set $GEM_* vars in that case, but @postmodern had some concerns about that in #410 (comment):

Furthermore, there is value in keeping user-installed gems separate from the ruby, in case you need to delete/re-install one or the other.

@postmodern @havenwood What do you think?
I would like to have this available in chruby 1.0.0, I'm happy to make a PR.

eregon commented

Proof-of-concept PR for $CHRUBY_SET_GEM_HOME: #423

postmodern commented

The reason why chruby sets GEM_HOME for non-root users is to protect the ruby's built-in gem home, which may not always be writable. This also helps keep the ruby's built-in gems from any user installed gems. Perhaps better logic would be to check if the ruby's GEM_HOME is writable? Another idea might be to split chruby_use into a lower-level function that only sets the ruby, which you could explicitly call instead of the main chruby function.

Also, for the record, I'm adverse to adding yet more CHRUBY_ env variables to user's shells.

eregon commented

The reason why chruby sets GEM_HOME for non-root users is to protect the ruby's built-in gem home, which may not always be writable.

Right, I know. This is very inconvenient when actually developing a Ruby implementation though, as it prevents using the current Ruby selected with chruby and the Ruby being developed without switching everytime when doing anything using gems.
(#410 (comment) for another way to say that)

This also helps keep the ruby's built-in gems from any user installed gems. Perhaps better logic would be to check if the ruby's GEM_HOME is writable?

That would not keep built-in and user-installed gems separated, so I (maybe wrongly?) assumed it's not an approach you wanted to take.
I think it could actually be nice feature and work well, so if you're not against it, I'm happy to make a PR for that approach.
Then chruby would only set GEM_HOME when it's needed (i.e., when gems otherwise would need sudo to be installed).

Another idea might be to split chruby_use into a lower-level function that only sets the ruby, which you could explicitly call instead of the main chruby function.

I'd still want chruby MYRUBY to work and not set GEM_HOME.

Also, for the record, I'm adverse to adding yet more CHRUBY_ env variables to user's shells.

@postmodern What other solution do you think is better for my use case presented in this issue?

eregon commented

@postmodern Any thoughts on my reply?

I like the idea of detecting if GEM_HOME is writable, it sounds like a much cleaner check than "is root" or "env var", should I proceed with that and make a PR then if you're not against it?

eregon commented

@postmodern Here is a PR implementing the idea of setting GEM_HOME only if the default gem directory is not writable. It looks very elegant to me, please review it: #431

dentarg commented

Due to this, I've been forced to switch to @eregon's fork of chruby that have #431: https://github.com/eregon/chruby/tree/do-no-set-gem-home

It was needed because I have a computer with Apple silicon (M2), and I want to be able run Ruby apps either on arm64 or x86_64 ("Intel").

Using ruby-build it was very easy to install Rubies for different architectures to different directories (and configure your shell for it), but I can't have them install native extensions (like for Nokogiri) to the same directory (GEM_HOME)

arm64 $ env | grep GEM
GEM_ROOT=/Users/dentarg/.arm64_rubies/2.7.6/lib/ruby/gems/2.7.0
GEM_HOME=/Users/dentarg/.gem/ruby/2.7.6
GEM_PATH=/Users/dentarg/.gem/ruby/2.7.6:/Users/dentarg/.arm64_rubies/2.7.6/lib/ruby/gems/2.7.0

intel $ env | grep GEM
GEM_ROOT=/Users/dentarg/.rubies/2.7.6/lib/ruby/gems/2.7.0
GEM_HOME=/Users/dentarg/.gem/ruby/2.7.6
GEM_PATH=/Users/dentarg/.gem/ruby/2.7.6:/Users/dentarg/.rubies/2.7.6/lib/ruby/gems/2.7.0

As an example, before switching to the chruby fork (using chruby 0.3.9 installed from Homebrew), I did bundle install for my app on arm64 (ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [arm64-darwin21]), then I also did bundle install for it on Intel (ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-darwin21]). When I tried to run my tests on arm64 I got this back:

arm64 $ b e rake
Traceback (most recent call last):
  13: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `<main>'
  12: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `select'
  11: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `block in <main>'
  10: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `require'
   9: from /Users/dentarg/starkast/wikimum/test/unit/markup_test.rb:4:in `<top (required)>'
   8: from /Users/dentarg/starkast/wikimum/test/unit/markup_test.rb:4:in `require_relative'
   7: from /Users/dentarg/starkast/wikimum/lib/services/markup.rb:3:in `<top (required)>'
   6: from /Users/dentarg/starkast/wikimum/lib/services/markup.rb:3:in `require'
   5: from /Users/dentarg/.gem/ruby/2.7.6/gems/html-pipeline-2.14.2/lib/html/pipeline.rb:3:in `<top (required)>'
   4: from /Users/dentarg/.gem/ruby/2.7.6/gems/html-pipeline-2.14.2/lib/html/pipeline.rb:3:in `require'
   3: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri.rb:10:in `<top (required)>'
   2: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri.rb:10:in `require_relative'
   1: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:7:in `<top (required)>'
/Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:7:in `require_relative': cannot load such file -- /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/2.7/nokogiri (LoadError)
  14: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `<main>'
  13: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `select'
  12: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `block in <main>'
  11: from /Users/dentarg/.gem/ruby/2.7.6/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `require'
  10: from /Users/dentarg/starkast/wikimum/test/unit/markup_test.rb:4:in `<top (required)>'
   9: from /Users/dentarg/starkast/wikimum/test/unit/markup_test.rb:4:in `require_relative'
   8: from /Users/dentarg/starkast/wikimum/lib/services/markup.rb:3:in `<top (required)>'
   7: from /Users/dentarg/starkast/wikimum/lib/services/markup.rb:3:in `require'
   6: from /Users/dentarg/.gem/ruby/2.7.6/gems/html-pipeline-2.14.2/lib/html/pipeline.rb:3:in `<top (required)>'
   5: from /Users/dentarg/.gem/ruby/2.7.6/gems/html-pipeline-2.14.2/lib/html/pipeline.rb:3:in `require'
   4: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri.rb:10:in `<top (required)>'
   3: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri.rb:10:in `require_relative'
   2: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:4:in `<top (required)>'
   1: from /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:30:in `rescue in <top (required)>'
/Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/extension.rb:30:in `require': dlopen(/Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/nokogiri.bundle, 0x0009): tried: '/Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/nokogiri.bundle' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e))) - /Users/dentarg/.gem/ruby/2.7.6/gems/nokogiri-1.13.6/lib/nokogiri/nokogiri.bundle (LoadError)
rake aborted!

I know chruby 1.0.0 will bring changes to GEM_HOME (#419) but I don't think using basename is enough, as it will be the same basename regardless of the architecture.

  • 👍1
eregon commented

@dentarg Indeed, this is a clear real-world example of why sharing gem homes is problematic and AFAIK the only completely safe and solution is to just let gems be installed inside that Ruby's prefix, i.e., the default, which chruby unfortunately overrides. I am thinking to make my own Ruby switcher based on chruby for that, but I'm not sure I have the time.

eregon commented

@postmodern It has been 3 years since #419.
In that time I had dozens of people mentioning this problem to me, and the only workaround for now is to use my fork's branch (https://github.com/eregon/chruby/tree/do-no-set-gem-home), which is of course inconvenient.
I know for instance Shopify's tooling uses chruby and some people there get this issue almost daily and have to workaround with unset GEM_HOME GEM_PATH (because chruby is pre-installed).

Do you plan to release chruby 1.0.0 one day?
I'm happy to help, if that means we release it sooner.

This bug existing since 3 years and having no release fixing it seems to clearly indicate chruby is completely unmaintained. This is very unfortunate given how widely chruby is used.
So let's fix it, whether it's #419, #431 or something based on it.

What are your concerns?
That chruby users using CRuby have to reinstall gems?
We can probably avoid that if that's the main issue.

postmodern commented

@eregon I am currently busy working on another large refactoring project, before I can circle back around and work on chruby. I have not yet heard of other people having this issue besides you or other truffleruby developers. I am just curious, if this is such a critical issue, why can't truffleruby just workaround the problem?

Currently, I do not plan on optionally not setting GEM_HOME if the ruby's gem-root is writable, since like I said before it would lead to gems being potentially installed in two separate locations based on where or how the ruby was installed, and that could lead to confusion. I do however plan on adding additional function hooks to allow configuring how chruby sets GEM_HOME or GEM_PATH. These changes would be incremental and could be released before 1.0.0.

eregon commented

I have not yet heard of other people having this issue besides you or other truffleruby developers.

For instance:

I am just curious, if this is such a critical issue, why can't truffleruby just workaround the problem?

TruffleRuby already spent significant amount of effort to workaround this chruby bug.
Notably there is an ABI check when loading C extensions, so at least it fails early.

What workaround are you thinking of? I don't think truffleruby can workaround chruby setting the GEM_HOME incorrectly.
Whoever sets the GEM_HOME is responsible to set it correctly (or not set it), so that's chruby, isn't it?

  • 👍1
eregon commented

I do however plan on adding additional function hooks to allow configuring how chruby sets GEM_HOME or GEM_PATH. These changes would be incremental and could be released before 1.0.0.

That would be nice. The defaults needs to change though, because the current GEM_HOME/GEM_PATH set by chruby are incorrect for all dev Rubies, and all non-CRuby. In other words, it only works for CRuby releases, in the case they are always built with the same flags (e.g., --enable-shared) and arch for a given release version on the same machine. It also fails for CRuby releases on platforms with multiple archs as said in #422 (comment) and postmodern/ruby-install#413 (comment).

  • 👍1
noahgibbs commented

I can verify that in my work on the YJIT team, and especially for speed.yjit.org, we have difficulties when shared dirs contain built native extensions. We handle this by deleting all gems in all shared directories on every build, as well as all built Ruby dirs we'll be using for that build. In general, my experience is that Rubygems/Bundler/etc deal poorly with trying to match up specific built native extensions to specific Rubies. So for benchmarking we use a Big Hammer to handle the situation: we delete everything, so there is clearly nothing stale or inappropriately shared.

This problem isn't unique to chruby, though we do have it when using chruby. We're a hard case - we build a lot of prerelease Rubies that all have the same version string, so RUBY_VERSION checking does nothing for us. We're prone to get crashes, slowdowns and other problems very regularly if we leave built gems sitting around. So we basically treat shared-across-multiple-Rubies gem dirs as a bug, and delete their contents when there's reason to care.

To preempt the same question: we do indeed work around the problem, by deleting everything. We need a fresh gem install of all native extensions for every build regardless, so saving old copies wouldn't do us any good. If we wanted shorter runs that didn't take multiple hours, or to reuse the same prerelease Ruby for multiple runs, we'd work around it by fiddling with dirs like GEM_HOME every time we changed versions. But if we need to manually manage our gem-related env vars every time we switch Rubies, chruby becomes a much less valuable tool.

  • 👍1
dentarg commented

I have not yet heard of other people having this issue besides you or other truffleruby developers.

?? – I'm no truffleruby developer :) Just a developer doing things with Ruby.

I've been a devote and happy chruby user for 10 years (yes, since start almost), but sadly, I no longer recommend friends and co-workers to use the released version of chruby. I now recommend people using eregon's branch. It would be great to change that.

  • 👍1
postmodern commented

As a compromise and to get the changes released faster, I am thinking of adding a separate dev.sh file which can be loaded along with chruby.sh and that changes how GEM_HOME/GEM_PATH are set, in order to support testing rubies of the same version but with different configurations.

As per @dentarg's use case of wanting to switch between aarm64 and x86-64 on Apple Silicon, I could also add a multi_arch.sh file which takes into account the current architecture.

Testing the same ruby version with different configurations or switching between aarch64 and x86-64 on specific hardware are niche use-cases vs. your more common Ruby app development. Providing opt-in solutions seems like a good compromise. Depending on whether these opt-in solutions become popular, they might become the default behavior in 1.0.0.

  • 👍3
SampsonCrowley commented

Testing the same ruby version with different configurations or switching between aarch64 and x86-64 on specific hardware are niche use-cases vs. your more common Ruby app development. Providing opt-in solutions seems like a good compromise. Depending on whether these opt-in solutions become popular, they might become the default behavior in 1.0.0.

I have to disagree that multiple architechtures are niche use cases. there are searches for it all over the web since apple silicone and rosetta have become main staple development machines. it has become a necessity even for many

  • 👍1
k0kubun commented

I often invoke a ruby command from another ruby for benchmark reasons, which results in pointing to a wrong GEM_PATH in the child process. This resulted in workarounds for benchmark scripts like Shopify/yjit-bench#138 and benchmark-driver/benchmark-driver@1938655, but it's still not ideal because it would not pick up gems installed under a normal environment.

In addition, I have the same problem as eregon; I develop ruby itself. I often have multiple installations of the same ruby version with different build flags. When build flags are different, I want them to not share the same extensions. I want to unset GEM_HOME for that reason as well.

All in all, if you contribute to or benchmark a ruby implementation and install that under a writable directory, you shouldn't set GEM_HOME or GEM_PATH. However, I understand that most users are not interested in developing/benchmarking ruby and some users install ruby to a non-writable directory, so I'd be happy with that feature being optional. I filed #487 as per #422 (comment).

k0kubun commented

By the way, I was using master branch of chruby, which is why I didn't notice 1.0.0 branch has #419. It does address the latter concern in my comment, but the former concern remains. So it'd be nice to have a way to keep GEM_HOME unset.

eregon commented

TBH I'm really annoyed about this, I spent a large amount of time to try to fix this in a way that pleases @postmodern in chruby, spent time to discuss the proper solution in RubyGems (where the general agreement has become to use the default gem home unless it's not writable, exactly the same behavior I proposed for chruby in #431), but my PR is just hidden on a branch and not the best approach (which is to not set GEM_HOME) and chruby feels very much unmaintained. I think it's time I hard fork chruby and create a more sensible and actually maintained and sound Ruby switcher. The fact that chruby keeps an incorrect GEM_HOME for years without doing anything for its users seems unbelievable (also tracked as #451). The issue has been known for 4 years now.
#431 is a log of how frustrating it can be to contribute to chruby, I think it's clear to anyone. That's also I believe the best solution for this issue as it aligns with how RubyGems will behave.
Anyway, good luck with your PR @k0kubun and getting it released. It would be good to finally see some progress here so maybe I can forget about this terrible experience of contributing to chruby and this bug.

eregon commented

I am just curious, if this is such a critical issue, why can't truffleruby just workaround the problem?

TruffleRuby already spent significant amount of effort to workaround this chruby bug. Notably there is an ABI check when loading C extensions, so at least it fails early.

BTW, this ABI check which now exists in TruffleRuby and CRuby has caused various issues and complications. Was it not for chruby and this long-time bug we would probably not need this ABI check. So while that's not chruby's direct doing, it's partly a consequence of not fixing this bug.

postmodern commented

@eregon like we have discussed on multiple occasions, changing the path of the GEM_HOME is a breaking change and would cause chruby users to suddenly lose all of their installed gems after upgrading. Since chruby is widely used this would cause a great deal of confusion and frustration, so I decided to push that change back to 1.0.0 where we could safely break with backwards compatibility. Likewise, rubygems also has to maintain backwards compatibility and not change default behavior too much, otherwise that could possibly cause issues with downstream users and Linux distributions which expect gems to be installed into specific directories.

I would have accepted #451, however I am very hesitant about adding additional bifurcating logic. I could envision scenarios where you are debugging an issue for a use and you need to determine if the gems are being installed into the correct location, so you then have to determine if the ruby's gem directory is writable or not in order to determine if chruby is going to use the ruby's gem dir or ~/.gem/.... This would likely result in more debugging, troubleshooting, and complexity.

Since chruby is loaded into user's shells, and because it has to run under Bash 3+ and Zsh, I have to be very cautious about changing any functionality and debate every change, especially those that might break or change default behavior for users. chruby also has an explicit policy of not accepting workaround fixes for upstream issues, which means I scrutinize issues and whether they could be better solved by rubygems or upstream Ruby. I also have to balance the needs of regular users vs the more exotic edge-cases which Ruby maintainers discover while testing Rubies. chruby is not an easy project to contribute to and requires a great deal of patience and compromising.

I am now considering a different approach to handling GEM_HOME. Extracting the logic which sets GEM_HOME and GEM_PATH out as an additional function hook that could be overrode by additional opt-in configuration files much like how auto.sh is implemented. Additionally, we could extract that logic entirely into an additional gem_home.sh file and make isolating gems in ~/.gem/rubies/$ruby/ opt-in. This would provide an absolute bare minimum user experience of just switching the rubies, but not setting GEM_HOME by default. This would work seamlessly for users who have all rubies installed into ~/.rubies; which means their ruby gem directories are writable by default. If the GEM_HOME code is extracted entirely into an opt-in gems.sh or gem_home.sh file, and users have rubies installed into /opt/rubies (such as myself) , those users would then need to decide if they want gem isolated in ~/.gem/rubies/$ruby/ or allow rubygems to pick the gem installation directory for the user; this could potentially mean rubygems install gems of different /opt/rubies rubies into ~/.gem, but maybe some users might want that? By moving the logic out into functions, this opens the door for customization and different opt-in configurations.

Example Code

# potentially defined in a `gems.sh` or `gem_home.sh` file
function chruby_gems_set()
{
		export GEM_HOME="$HOME/.gem/$RUBY_ENGINE/$RUBY_VERSION" # could be extracted into another function in case people want to configure the `GEM_HOME` template string
		export GEM_PATH="$GEM_HOME${GEM_ROOT:+:$GEM_ROOT}${GEM_PATH:+:$GEM_PATH}"
		export PATH="$GEM_HOME/bin:$PATH"
}

# potentially defined in a `gems.sh` or `gem_home.sh` file
function chruby_post_hook() { chruby_set_gem_home }

function chruby_set()
{
    # add the $ruby/bin directory to PATH and query the ruby's information
    # ...
    chruby_post_hook
}

Sorry for not working on the 1.0.0 branch. I have been extremely busy over the last four-six years, with both commercial work (2014-2020) and other Open Source work (2020-2023). I will start working on 1.0.0 again; I just added relisting of rubies directories.

  • 👍2
k0kubun commented

FYI: #487 is now merged to 0.4.0 branch.