Undefined method '[]' for nil:NilClass when running two provisioning recipes in one chef-client session
Closed this issue · 1 comments
Hey team! I'm still running into undefined method '[]' for nil:NilClass, and it's preventing me from doing any meaningful provisioning. I've been battling that and Net::SSH::Exception: remote forwarding request failed (at this point I have bailed from using anything but individual machine resources to try to avoid the Net::SSH errors).
I'm using the vagrant provider with vmware fusion. I've simplified my recipes as much as possible. Everything appears to be up to date (chef-provisioning-vagrant is 0.8.1 and chef provisioning is 0.17). The error seems to consistently happen on second run. After running an initial converge (or setup) with blank recipes, the second time a machine runs (for instance, converges with additional recipes) it fails. The debug output pegs the error here:
[2014-12-19T06:49:09-08:00] DEBUG: NoMethodError: machine[database] (@recipe_files::/Users/adam/Development/chef/provisioning/test.local.converge.rb line 14) had an error: NoMethodError: undefined method `[]' for nil:NilClass
/Users/adam/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vagrant-0.8.1/lib/chef/provisioning/vagrant_driver/driver.rb:462:in `strip_quotes'
/Users/adam/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vagrant-0.8.1/lib/chef/provisioning/vagrant_driver/driver.rb:431:in `create_ssh_transport'
/Users/adam/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vagrant-0.8.1/lib/chef/provisioning/vagrant_driver/driver.rb:391:in `transport_for'
/Users/adam/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vagrant-0.8.1/lib/chef/provisioning/vagrant_driver/driver.rb:368:in `machine_for'
/Users/adam/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vagrant-0.8.1/lib/chef/provisioning/vagrant_driver/driver.rb:74:in `ready_machine'
/Users/adam/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-0.17/lib/chef/provider/machine.rb:34:in `block in <class:Machine>'
/opt/chefdk/embedded/apps/chef/lib/chef/provider/lwrp_base.rb:60:in `instance_eval'
/opt/chefdk/embedded/apps/chef/lib/chef/provider/lwrp_base.rb:60:in `recipe_eval_with_update_check'
/opt/chefdk/embedded/apps/chef/lib/chef/provider/lwrp_base.rb:45:in `block in action'
/Users/adam/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-0.17/lib/chef/provider/machine.rb:51:in `block in <class:Machine>'
/opt/chefdk/embedded/apps/chef/lib/chef/provider/lwrp_base.rb:60:in `instance_eval'
/opt/chefdk/embedded/apps/chef/lib/chef/provider/lwrp_base.rb:60:in `recipe_eval_with_update_check'
/opt/chefdk/embedded/apps/chef/lib/chef/provider/lwrp_base.rb:45:in `block in action'
/opt/chefdk/embedded/apps/chef/lib/chef/provider.rb:121:in `run_action'
/opt/chefdk/embedded/apps/chef/lib/chef/resource.rb:648:in `run_action'
/opt/chefdk/embedded/apps/chef/lib/chef/runner.rb:49:in `run_action'
/opt/chefdk/embedded/apps/chef/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chefdk/embedded/apps/chef/lib/chef/runner.rb:81:in `each'
/opt/chefdk/embedded/apps/chef/lib/chef/runner.rb:81:in `block in converge'
/opt/chefdk/embedded/apps/chef/lib/chef/resource_collection.rb:98:in `block in execute_each_resource'
/opt/chefdk/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chefdk/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chefdk/embedded/apps/chef/
...so I checked into the vagrant ssh-config that it's running, and I got this output
The provider for this Vagrant-managed machine is reporting that it
is not yet ready for SSH. Depending on your provider this can carry
different meanings. Make sure your machine is created and running and
try again. Additionally, check the output of `vagrant status` to verify
that the machine is in the state that you expect. If you continue to
get this error message, please view the documentation for the provider
you're using.
It makes sense that the string index fails since the regexp at driver.rb:431 is not designed to capture anything from this output (though, on an unrelated note, it might be good to guard against vagrant returning unexpected messages by passing them back to the user if they don't meet the expected format...)
A few interesting details: After the first run, vagrant ssh-config works for all machines. If I run the second half with a separate chef-client -z call, everything appears to work fine. However, if I execute both of the recipes sequentially with chef-client -z provisioning/test.local.setup.rb provisioning/test.local.converge.rb, then I get the error consistently. On exit, the first node in the second run becomes unresponsive to vagrant ssh and the vagrant ssh-config command returns the result posted above. So, as far as I've been able to tell, it appears that something about having a second run in the same chef-client session is breaking ssh for the first machine that it works with. Sample recipes follow below.
The machines are all interdependent and firewalled to the hilt, so I need to provision the machines this way so that the IP addresses are all available before the main converge happens. This is, as far as I'm aware, the best way to quickly get a complex, interdependent, and dynamically configuring set of vms prepped for integration testing via chef.
I'm open to the possibility that I am doing something terribly wrong, but I've read every doc I can find as completely as possible and am not coming up with anything. Please tell me if there is anything in here that is not best practice!
Thanks!
# This recipe runs first
require 'chef/provisioning'
require 'chef/provisioning/vagrant_driver'
with_driver 'vagrant'
vagrant_cluster '/Users/adam/.chef/vms/'
vagrant_box 'ubuntu-1404-64' do
url 'https://opscode-vm-bento.s3.amazonaws.com/vagrant/vmware/opscode_ubuntu-14.04_chef-provisionerless.box'
end
with_machine_options :vagrant_options => {
'vm.box' => 'ubuntu-1404-64',
}
machine 'database' do
chef_environment 'integration_testing'
role 'database'
action :converge
end
machine "combo" do
chef_environment 'integration_testing'
role 'web'
role 'queue'
role 'render'
action :converge
end
machine "license" do
chef_environment 'integration_testing'
role 'license'
action :converge
end
machine "monitor" do
chef_environment 'integration_testing'
role 'monitor'
action :converge
end
# This recipe runs second
require 'chef/provisioning'
require 'chef/provisioning/vagrant_driver'
with_driver 'vagrant'
vagrant_cluster '/Users/adam/.chef/vms/'
vagrant_box 'ubuntu-1404-64' do
url 'https://opscode-vm-bento.s3.amazonaws.com/vagrant/vmware/opscode_ubuntu-14.04_chef-provisionerless.box'
end
with_machine_options :vagrant_options => {
'vm.box' => 'ubuntu-1404-64',
}
machine_batch do
action :converge
['database', 'combo', 'license', 'monitor'].each do |machine_name|
machine machine_name do
recipe 'mycookbook::run_all'
end
end
end
After seeing ongoing strange behavior, and to control as many variables as possible, I ended up blowing away my vagrant installation and all cached data. I am currently seeing more stable results. I'll report back on this issue if I see the behavior return. Thanks!