bazel-contrib/rules_ruby

Document how to use "pre-built binary" gems to avoid install-time non-hermeticity

Closed this issue · 3 comments

alexeagle commented

The psych gem (version 5.1.1.1) fails to install on my machine, log contains:

conftest.c:3:10: fatal error: yaml.h: No such file or directory
    3 | #include <yaml.h>
      |          ^~~~~~~~
compilation terminated.
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <yaml.h>
/* end */

As documented here, for example
https://github.com/sparklemotion/nokogiri#native-gems-faster-more-reliable-installation
there's a way to start from a pre-compiled native gem instead. I think most Bazel users should be using this to achieve reproducible builds.

(Of course, you could also imagine a cc_library to build the C extension, but then we're swapping out part of the bundle installer which is a bigger project)

p0deje commented

As documented here, for example
https://github.com/sparklemotion/nokogiri#native-gems-faster-more-reliable-installation
there's a way to start from a pre-compiled native gem instead. I think most Bazel users should be using this to achieve reproducible builds.

That's only available for certain gems like nokogiri. For pysch, we could potentially install libyaml and follow the steps in https://github.com/ruby/psych#installation to point to use it instead of system libyaml.

How does other rules (node/python) handle installation that expects certain libraries in the system?

alexeagle commented

Node.js and Python rules do a poor job of this as well. You have three choices:

  1. custom wiring with cc_library to replace the package manager's build system with Bazel, and figure out how to stitch the result into how the ruleset gets its third-party libraries. Pretty much no one knows how to do this although it feels ilke the principled answer
  2. just let the package manager's build system do its thing and deal with non-hermeticity. 90% of bazel users do this.
  3. use a pre-compiled binary package for your platform. In cases where upstream maintainers don't ship one, build one yourself and put into artifactory or some custom registry. The sophisticated 10% do this

Random note, the native build system used under Node.js is GYP and they considered using Bazel to implement it at one point

p0deje commented

If you're on a supported platform, either gem install or bundle install should install a native gem without any additional action on your part. This installation should only take a few seconds, and your output should look something like:

Native gems are selected during installation time automatically by RubyGems. We can't really force that. Let's for now stick to option 2.