How does one use Heroku-20 stack for local development WITH the same ruby/bundler versions as Heroku defaults to
Closed this issue · 5 comments
Where is it documented how to get the latest heroku:20-build docker image to that happy place for local Dockerized Rails development where it has the default/recommended ruby, bundler versions (at a minimum, and probably also nodejs, npm, and yarn versions)?
I think I understand that "buildpacks" are the magic sauce that lets Heroku support such a wide range of frameworks.
And I think I understand that these stack images are kind of a foundation that those buildpacks rest upon.
Yet, the Heroku online service does have a 'default' ruby version (currently 2.7.3?). And when we do a git push, Heroku only uses its 'curated' Bundler version (2.2.16?). And I expect (especially for Rails 6 apps) there are particular recommended versions of nodejs, npm, and Yarn, too.
After a couple of years of "trial and error" on Heroku-18, trying to iteratively zero in on reasonable dev/prod parity in our local Docker Rails development setup, I figured I'd just ask... **how/where do we 'get' the current-default ruby version out of the correct ruby buildpack? **
I assume that in a Dockerfile, right below FROM heroku/heroku:20-build developers need add a series of apt-get X Y Z commands, and probably purging and reinstalling the default version of Ruby (from where?) and probably manual installation of the "current correct" Bundler version (trivially easy except - which version?)
Is that "default ruby version + correct bundler version" Dockerfile documented anywhere? (This git repo seems the logical place.)
If not, oughtn't it be clearly documented given Heroku (correctly) stresses the importance of dev/prod parity?
A few notes:
Running your latest heroku/heroku:20-build via docker run --rm -v ${PWD}:/usr/src -w /usr/src -ti heroku/heroku:20-build /bin/bash I note that:
- "ruby -v" shows ruby 2.7.0p0 (but I think the the current default is 2.7.3)
This is how we (successfully) could update the obsolete stack image Ruby version back on heroku-18:
When attempting to update Ruby from a heroku buildpack, what USED to work (in heroku:18-build) is:
# Dockerfile (just seeing if ruby can be updated to a later Heroku-supplied version)
RUN apt-get update -qq && apt-get install -y nodejs
RUN apt remove -y --purge ruby && curl -s --retry 3 -L https://heroku-buildpack-ruby.s3.amazonaws.com/heroku-18/ruby-2.5.8.tgz | tar -xz
RUN ruby -v
RUN gem list
That initial snippet successfully replaced the default image Ruby version with a later version, obtained right from one of your buildpacks. The output of running docker build --progress=plain --no-cache -t hiapptmp . is:
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.40kB done
#2 DONE 0.0s
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#3 [internal] load metadata for docker.io/heroku/heroku:18-build
#3 DONE 0.0s
#4 [1/5] FROM docker.io/heroku/heroku:18-build
#4 resolve docker.io/heroku/heroku:18-build done
#4 DONE 0.0s
#5 [2/5] RUN apt-get update -qq && apt-get install -y nodejs
#5 7.257 Reading package lists...
.... bunch of messages
#5 DONE 13.1s
#6 [3/5] RUN apt remove -y --purge ruby && curl -s --retry 3 -L https://her...
#6 0.210
#6 0.210 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
#6 0.210
#6 0.233 Reading package lists...
#6 1.394 Building dependency tree...
#6 1.586 Reading state information...
#6 1.735 The following packages were automatically installed and are no longer required:
#6 1.735 ruby-did-you-mean ruby-minitest ruby-net-telnet ruby-power-assert
#6 1.735 rubygems-integration
#6 1.736 Use 'apt autoremove' to remove them.
#6 1.792 The following packages will be REMOVED:
#6 1.793 libruby2.5* rake* ruby* ruby-dev* ruby-test-unit* ruby2.5* ruby2.5-dev*
#6 2.039 0 upgraded, 0 newly installed, 7 to remove and 6 not upgraded.
#6 2.039 After this operation, 15.2 MB disk space will be freed.
(Reading database ... 36188 files and directories currently installed.)
#6 2.085 Removing ruby-dev:amd64 (1:2.5.1) ...
#6 2.104 Removing ruby2.5-dev:amd64 (2.5.1-1ubuntu1.9) ...
#6 2.143 Removing ruby (1:2.5.1) ...
#6 2.161 Removing ruby2.5 (2.5.1-1ubuntu1.9) ...
#6 2.182 Removing libruby2.5:amd64 (2.5.1-1ubuntu1.9) ...
#6 2.270 Removing ruby-test-unit (3.2.5-1) ...
#6 2.293 Removing rake (12.3.1-1ubuntu0.1) ...
#6 2.335 Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
#6 DONE 5.5s
#7 [4/5] RUN ruby -v
#7 0.211 ruby 2.5.8p224 (2020-03-31 revision 67882) [x86_64-linux]
#7 DONE 0.2s
#8 [5/5] RUN gem list
#8 0.438 bigdecimal (default: 1.3.4)
#8 0.438 cmath (default: 1.0.0)
#8 0.438 csv (default: 1.0.0)
#8 0.438 date (default: 1.0.0)
#8 0.438 dbm (default: 1.0.0)
#8 0.438 did_you_mean (1.2.0)
#8 0.438 etc (default: 1.0.0)
#8 0.438 fcntl (default: 1.0.0)
#8 0.438 fiddle (default: 1.0.0)
#8 0.438 fileutils (default: 1.0.2)
#8 0.438 gdbm (default: 2.0.0)
#8 0.438 io-console (default: 0.4.6)
#8 0.438 ipaddr (default: 1.2.0)
#8 0.438 json (default: 2.1.0)
#8 0.438 minitest (5.10.3)
#8 0.438 net-telnet (0.1.1)
#8 0.438 openssl (default: 2.1.2)
#8 0.438 power_assert (1.1.1)
#8 0.438 psych (default: 3.0.2)
#8 0.438 rake (12.3.3)
#8 0.438 rdoc (default: 6.0.1.1)
#8 0.438 scanf (default: 1.0.0)
#8 0.438 sdbm (default: 1.0.0)
#8 0.438 stringio (default: 0.0.1)
#8 0.438 strscan (default: 1.0.0)
#8 0.438 test-unit (3.2.7)
#8 0.438 webrick (default: 1.4.2)
#8 0.438 xmlrpc (0.3.0)
#8 0.438 zlib (default: 1.0.0)
#8 DONE 0.5s
#9 exporting to image
#9 exporting layers
#9 exporting layers 0.4s done
#9 writing image sha256:734b82c8ef7738c933b92115133747f57183279fd4c19e8435322266b3b8888e done
#9 naming to docker.io/library/hiapptmp done
#9 DONE 0.4s
That method does not work on 20-build
Trying the same thing, using 20-build, and a ruby 2.7.3 does install the new ruby, but the installation is broken:
# Dockerfile (just seeing if ruby can be updated to a later Heroku-supplied version)
FROM heroku/heroku:20-build
RUN apt-get update -qq && apt-get install -y nodejs
RUN apt remove -y --purge ruby && curl -s --retry 3 -L https://heroku-buildpack-ruby.s3.amazonaws.com/heroku-20/ruby-2.7.3.tgz | tar -xz
RUN ruby -v
RUN gem list
$ docker build --progress=plain --no-cache -t hiapptmp .
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.66kB done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/heroku/heroku:20-build
#3 DONE 0.0s
#4 [1/5] FROM docker.io/heroku/heroku:20-build
#4 CACHED
#5 [2/5] RUN apt-get update -qq && apt-get install -y nodejs
#5 6.419 Reading package lists...
... bunch of messages
#5 DONE 12.1s
#6 [3/5] RUN apt remove -y --purge ruby && curl -s --retry 3 -L https://her...
#6 0.300
#6 0.300 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
#6 0.300
#6 0.322 Reading package lists...
#6 1.144 Building dependency tree...
#6 1.309 Reading state information...
#6 1.453 The following packages were automatically installed and are no longer required:
#6 1.453 ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit ruby-xmlrpc
#6 1.454 rubygems-integration
#6 1.454 Use 'apt autoremove' to remove them.
#6 1.501 The following packages will be REMOVED:
#6 1.502 libruby2.7* rake* ruby* ruby-dev* ruby2.7* ruby2.7-dev*
#6 1.750 0 upgraded, 0 newly installed, 6 to remove and 6 not upgraded.
#6 1.750 After this operation, 19.2 MB disk space will be freed.
(Reading database ... 38839 files and directories currently installed.)
#6 1.800 Removing ruby-dev:amd64 (1:2.7+1) ...
#6 1.824 Removing ruby2.7-dev:amd64 (2.7.0-5ubuntu1.4) ...
#6 1.868 Removing ruby2.7 (2.7.0-5ubuntu1.4) ...
#6 1.892 Removing libruby2.7:amd64 (2.7.0-5ubuntu1.4) ...
#6 2.033 Removing rake (13.0.1-4) ...
#6 2.061 Removing ruby (1:2.7+1) ...
#6 2.102 Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
#6 DONE 12.5s
#7 [4/5] RUN ruby -v
#7 0.253 container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory"
#7 ERROR: executor failed running [/bin/sh -c ruby -v]: runc did not terminate sucessfully
------
> [4/5] RUN ruby -v:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c ruby -v]: runc did not terminate sucessfully
setting WORKDIR to /usr gets "closer" (ruby -v works) but Gem systems gets bolluxed
# Dockerfile
FROM heroku/heroku:20-build
WORKDIR /usr
RUN apt-get update -qq && apt-get install -y nodejs
RUN apt remove -y --purge ruby && curl -s --retry 3 -L https://heroku-buildpack-ruby.s3.amazonaws.com/heroku-20/ruby-2.7.3.tgz | tar -xz
RUN ruby -v
RUN gem list
docker build --progress=plain --no-cache -t hiapptmp .
... first part is same
.... but now the ruby -v works:
#8 [5/6] RUN ruby -v
#8 0.276 ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [x86_64-linux]
#8 DONE 0.3s
... however Gem system broken....
#9 [6/6] RUN gem list
#9 0.228 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:50:in `<class:Specification>': undefined method `rubyforge_project=' for class `Gem::Specification' (NameError)
#9 0.229 from /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:49:in `<top (required)>'
#9 0.229 from /usr/lib/ruby/2.7.0/rubygems.rb:1427:in `require'
#9 0.229 from /usr/lib/ruby/2.7.0/rubygems.rb:1427:in `<top (required)>'
#9 0.229 from <internal:gem_prelude>:1:in `require'
#9 0.229 from <internal:gem_prelude>:1:in `<internal:gem_prelude>'
#9 ERROR: executor failed running [/bin/sh -c gem list]: runc did not terminate sucessfully
------
> [6/6] RUN gem list:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c gem list]: runc did not terminate sucessfully
another possible issue for build-20 related to Rubygems?
Also disconcerting (with the -20:build) is some possible issue related to rubygems out of the box... I assume a simple gem update --system ought to work out-of-the-box on the stack image?
docker run --rm -v ${PWD}:/usr/src -w /usr/src -ti heroku/heroku:20-build /bin/bash
# gem update --system --no-document
Updating rubygems-update
Fetching rubygems-update-3.2.17.gem
Successfully installed rubygems-update-3.2.17
Installing RubyGems 3.2.17
Traceback (most recent call last):
5: from setup.rb:23:in `<main>'
4: from setup.rb:23:in `require'
3: from /var/lib/gems/2.7.0/gems/rubygems-update-3.2.17/lib/rubygems.rb:1348:in `<top (required)>'
2: from /var/lib/gems/2.7.0/gems/rubygems-update-3.2.17/lib/rubygems.rb:1348:in `require'
1: from /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:49:in `<top (required)>'
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:50:in `<class:Specification>': undefined method `rubyforge_project=' for class `Gem::Specification' (NameError)
(although updating to a specific version less than 3.1.4 does seem to work)
Hi!
I agree it would be helpful if there was a better story for this. Longer term Cloud Native Buildpacks are going to be the solution for this, since with them, one can run virtually an identical build locally as is run in production.
In the meantime I'd check out the Dockerflles in #56 - particularly the more recent comments - there's an example of using the Ruby buildpack to install Ruby.
Alternatively a pretty string option is to just use the official Docker Ruby images (much lighter-weight than the Heroku stack images for local development) and instead rely on Heroku CI/Review Apps/staging app to ensure correctness before production deployment. After all, there are many many differences running locally compared to production (the docker image is likely the least of it; differences in DB configuration/version, RAILS_ENV, ...) so there still needs to be validation performed on Heroku and not just locally.
When attempting to update Ruby from a heroku buildpack, what USED to work (in heroku:18-build) is:
Yeah I wouldn't do that. Use the whole Ruby buildpack and not just the Ruby binary archive. The archive is not meant to be extracted over the system packages. The Ruby buildpack installs the files into the app directory and adds them to PATH, GEM_PATH etc.
Thank you Ed. Can you offer a specific suggestion re "official Docker Ruby images" for Rails apps that will end up on Heroku? In https://hub.docker.com/_/ruby for example there are a lot of options: 2.7.3-buster, 2.7.3-slim-buster, 2.7.3-alpine3.13, 2.7.3-alpine3.12 which is Alpine and Debian, but Heroku uses Ubuntu, yes? and none of the options in that docker hub seem to be Ubuntu.
And of course then there is the issue of all the many apt-get packages to include. Any suggestions for figuring that out?
I suspect I'm not the only Heroku user who uses Heroku precisely so I can focus on my app instead of hundreds of pieces of underlying stack-related minutiae (well, plus the excellent devops of course). So it's unfortunate to discover that no, the Heroku stack images are not a way to quickly configure Docker so we can get back to app development.
It seems like "Sample Dockerfile for a Rails 6 app destined for Heroku-20" would be something Heroku could/should/would publish.
Many of our internal Ruby projects don't use docker for local development and instead run on the local machine. We then use Heroku CI/Review Apps/staging apps to catch any dev-prod parity issues. If you would prefer to use Docker locally, then I don't think it matters too much which base image you use. In terms of apt packages, just install the absolute minimum so that gems install (eg libpq if using the pg gem).
A local testing environment is never going to fully match a remote production environment, and of the things that differ, the fact that the Docker image OS is Debian vs Ubuntu is much less frequently going to be the cause compared to other factors (eg different env vars, different DB configuration/version/not full prod data, framework debug mode, different load balancer, not real production user load etc). And for the cases where it does matter - there's still Review Apps, staging apps etc.
There are obviously tradeoffs of all approaches, however I think usability locally can often be more important than trying to have still only partial prod-parity.
You are more experienced and probably correct re the tradeoffs of local stack simplicity vs dev/prod parity. However (and maybe because I'm an old guy) I've always leaned pretty hard towards "try to catch at least some of the 'big' gotchas as early as possible right on the dev machine."
That said, review apps and staging apps have been awesomely helpful (we use both). And even wrote up a small blog post favorably comparing the speed of Heroku CI vs rspec locally: https://blog.pardner.com/2019/01/making-heroku-ci-almost-as-fast-as-running-rspec-locally/