SSHKit::Runner::ExecuteError "rails exit status: 127" in versions 0.5 and 0.6

Question

SSHKit::Runner::ExecuteError "rails exit status: 127" in versions 0.5 and 0.6

Closed this issue 8 years ago · 11 comments

I recently upgraded from 0.4 to 0.6 and cap production db:pull no longer works. The problem seems to be in the code from PR #70 (it occurs in both 0.5 and 0.6, but 0.4 works fine) capistrano-db-tasks-0.6/lib/capistrano-db-tasks/database.rb:111 when its trying to execute this command on the remote side:

bundle exec rails runner "puts '__CAPISTRANODB_CONFIG_BEGIN_FLAG__' + ActiveRecord::Base.connection.instance_variable_get(:@config).to_yaml + '__CAPISTRANODB_CONFIG_END_FLAG__'" 2>/dev/null

I get an error:

cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing on host abcsite.com: rails exit status: 127
rails stdout: Nothing written
rails stderr: Nothing written
/Users/steven/.rvm/gems/ruby-2.3.1@diet/gems/sshkit-1.11.5/lib/sshkit/command.rb:100:in `exit_status='
/Users/steven/.rvm/gems/ruby-2.3.1@diet/gems/sshkit-1.11.5/lib/sshkit/backends/netssh.rb:151:in `execute_command'
/Users/steven/.rvm/gems/ruby-2.3.1@diet/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:141:in `block in create_command_and_execute'
/Users/steven/.rvm/gems/ruby-2.3.1@diet/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:141:in `tap'
/Users/steven/.rvm/gems/ruby-2.3.1@diet/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:141:in `create_command_and_execute'
/Users/steven/.rvm/gems/ruby-2.3.1@diet/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:60:in `capture'
/Users/steven/.rvm/gems/ruby-2.3.1@diet/gems/capistrano-db-tasks-0.5/lib/capistrano-db-tasks/database.rb:111:in `block (2 levels) in initialize'

My server is running RVM and (obviously) bundler, so I think it may be some kind of path or shell config problem, but I really don't know where to start.

I've checked my deploy.rb and everything is set like the README requests. I used to have the gem inside a group :development block, but I tried moving it out of that block so it would be present in all environments, but that didn't help:

# from Gemfile
group :development do
  gem 'capistrano',         require: false
  gem 'capistrano-rvm',     require: false
  gem 'capistrano-rails',   require: false
  gem 'capistrano-bundler', require: false
  gem 'capistrano3-puma',   require: false
end

gem "capistrano-db-tasks", '0.5', require: false

If I ssh into the server and directly run that command (preceding it with RAILS_ENV=production), it works and produces the following output

deploy@dabc1:~/apps/abc/current$ RAILS_ENV=production bundle exec rails runner "puts '__CAPISTRANODB_CONFIG_BEGIN_FLAG__' + ActiveRecord::Base.connection.instance_variable_get(:@config).to_yaml + '__CAPISTRANODB_CONFIG_END_FLAG__'"
__CAPISTRANODB_CONFIG_BEGIN_FLAG__---
:adapter: postgresql
:encoding: unicode
:pool: 5
:database: abc_production
:username: deploy
:password: ...pwdhere...
__CAPISTRANODB_CONFIG_END_FLAG__

My setup has ENV variables like DB password in a .sh file that gets loaded when an interactive shell loads.

Is anyone else having this problem? Any suggestions?

Here is a full trace (I added some puts to netssh.rb to I could see what command was failing).

steven$ cap production db:pull --trace
** Invoke production (first_time)
** Execute production
** Invoke load:defaults (first_time)
** Execute load:defaults
** Invoke bundler:map_bins (first_time)
** Execute bundler:map_bins
** Invoke deploy:set_rails_env (first_time)
** Execute deploy:set_rails_env
** Invoke deploy:set_linked_dirs (first_time)
** Execute deploy:set_linked_dirs
** Invoke deploy:set_rails_env
** Invoke rvm:hook (first_time)
** Execute rvm:hook
***************************
[ -d ~/.rvm ]
***************************
** Invoke rvm:check (first_time)
** Execute rvm:check
***************************
~/.rvm/bin/rvm version
***************************
rvm 1.27.0 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]
***************************
~/.rvm/bin/rvm current
***************************
ruby-2.3.1
***************************
~/.rvm/bin/rvm default do ruby --version
***************************
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
** Invoke db:pull (first_time)
** Invoke db:local:sync (first_time)
** Execute db:local:sync
00:00 db:local:sync
      Loading local database config
Local database: abc_development
Please enter answer (Are you sure you want to erase your local database with server database (y)es, (n)o  ? ): y
      Loading local database config
      Loading remote database config
***************************
if test ! -d /home/deploy/apps/abc/current; then echo "Directory does not exist '/home/deploy/apps/abc/current'" 1>&2; false; fi
***************************
***************************
bundle exec rails runner "puts '__CAPISTRANODB_CONFIG_BEGIN_FLAG__' + ActiveRecord::Base.connection.instance_variable_get(:@config).to_yaml + '__CAPISTRANODB_CONFIG_END_FLAG__'" 2>/dev/null
***************************
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing on host abcsite.com: rails exit status: 127
rails stdout: Nothing written
rails stderr: Nothing written
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
SSHKit::Command::Failed: rails exit status: 127
rails stdout: Nothing written
rails stderr: Nothing written
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/command.rb:100:in `exit_status='
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/netssh.rb:151:in `execute_command'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:141:in `block in create_command_and_execute'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:141:in `tap'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:141:in `create_command_and_execute'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:60:in `capture'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/capistrano-db-tasks-0.5/lib/capistrano-db-tasks/database.rb:111:in `block (2 levels) in initialize'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:93:in `with'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/capistrano-db-tasks-0.5/lib/capistrano-db-tasks/database.rb:110:in `block in initialize'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:85:in `within'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/capistrano-db-tasks-0.5/lib/capistrano-db-tasks/database.rb:109:in `initialize'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/capistrano-db-tasks-0.5/lib/capistrano-db-tasks/database.rb:203:in `new'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/capistrano-db-tasks-0.5/lib/capistrano-db-tasks/database.rb:203:in `remote_to_local'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/capistrano-db-tasks-0.5/lib/capistrano-db-tasks/dbtasks.rb:41:in `block (4 levels) in <top (required)>'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:29:in `instance_exec'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/backends/abstract.rb:29:in `run'
/Users/steven/.rvm/gems/ruby-2.3.1@abc/gems/sshkit-1.11.5/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => db:pull => db:local:sync

Answer 1 · 2017-01-02T04:27:03.000Z

@stevenchanin thanks a lot for detailed issues. I'll try to reproduce and fix it as soon as possible.

Answer 2 · 2017-01-04T20:19:14.000Z

I can confirm the same issue.

Answer 3 · 2017-01-05T14:45:10.000Z

@stevenchanin or @dmitry can you show your Capfile? Do you use rvm locally?

Try to add set :rvm_map_bins, %w{gem rake ruby rails bundle} to deploy.rb. Is it helpful?

Answer 4 · 2017-01-05T16:46:20.000Z

@numbata -- adding the set :rvm_map_bins fixed the problem.

Yes. I use rvm locally and on the remote side.

I tested your proposal by:

add the set :rvm_map_bins to my deploy.rb
updating my Gemfile to use 0.6 instead of 0.4. Doing a bundle install. I was able to db:pull successfully.
delete my gemset and completely rebuild it (that ensured that I didn't even have a copy of 0.4 anymore). I was able to db:pull successfully.

For reference, here is a gist of my deploy.rb with the new set command.
https://gist.github.com/stevenchanin/a9c474e1abdeaec9cf5d3d0f976b3e3b

Answer 5 · 2017-01-05T16:53:40.000Z

@stevenchanin thanks! This situation occurred due capistrano-rvm gem don't wrap rails command (prooflink). It's strange, because capistrano-bundler do.

And since Rails 5.0+ has rake commands built into the rails executable (link).

Answer 6 · 2017-01-05T17:02:50.000Z

Awesome debugging. That looks like it would have been hard to figure out. Nice work!

Answer 7 · 2017-01-05T17:06:47.000Z

@stevenchanin you should bump capistrano/rvm#56 😆

Answer 8 · 2017-01-06T20:18:34.000Z

@stevenchanin @dmitry can i close this pr now?

Answer 9 · 2017-01-06T20:47:55.000Z

@numbata thanks for your help. It works now.

Answer 10 · 2017-01-06T20:48:04.000Z

@numbata -- yes! Thanks!

Answer 11 · 2018-03-11T07:11:06.000Z

@numbata Thanks!