Cluster ssh/kick undefined method `each' for nil:NilClass (NoMethodError)
gwilton opened this issue · 5 comments
gwilton commented
I want to upgrade chef/ironfan and decided to try things out with a sandbox cluster. I am using the master version of ironfan_homebase which has ironfan 4.11.3 and it installs chef 10.30.2. I able to launch/bootstrap the cluster, but the cluster ssh or kick commands fail with the following exception, "./gems/chef-10.30.2/lib/chef/knife/ssh.rb:100:in block in session': undefined method each' for nil:NilClass (NoMethodError)".
Seems the issue is in the session method under chef/knife/ssh.rb. @action_nodes is coming back a nil. From the -VV output it seems all the node information is gathered and chef search node name:sandbox* is working as expected. Not sure what could be happening here. Can anyone help.
@action_nodes ends up being null.
def session
config[:on_error] ||= :skip
ssh_error_handler = Proc.new do |server|
if config[:manual]
node_name = server.host
else
@action_nodes.each do |n|
node_name = n if format_for_display(n)[config[:attribute]] == server.host
end
end
ironfan_homebase (master)]$ knife cluster show sandbox
Inventorying servers in sandbox cluster, all facets, all servers
sandbox: Loading chef
sandbox: Loading ec2
sandbox: Reconciling DSL and provider information
+------------------+-------+---------+----------+------------+-------------+------------+-------------+--------------+------------+
| Name | Chef? | State | Flavor | AZ | Env | MachineID | Public IP | Private IP | Created On |
+------------------+-------+---------+----------+------------+-------------+------------+-------------+--------------+------------+
| sandbox-simple-0 | yes | running | m1.large | us-east-1c | development | i-2f739c56 | 50.19.27.74 | 10.40.127.30 | 2013-12-19 |
+------------------+-------+---------+----------+------------+-------------+------------+-------------+--------------+------------+
ironfan_homebase (master)]$ knife cluster ssh sandbox uptime -VV
DEBUG: Using configuration from /Users/wilton/Documents/workspace/github/ironfan_homebase/.chef/knife.rb
Inventorying servers in sandbox cluster, all facets, all servers
INFO: Loading cluster /Users/wilton/Documents/workspace/github/ironfan_homebase/clusters/sandbox.rb
sandbox: Loading chef
sandbox: Loading ec2 sandbox: - loading clients
DEBUG: Signing the request as wilton
DEBUG: Sending HTTP Request via GET to chef-server.shrbt.com:4000//search/client
DEBUG: ---- HTTP Status and Header Data: ----
DEBUG: HTTP 1.1 200 OK
DEBUG: content-type: application/json; charset=utf-8
DEBUG: connection: close
DEBUG: server: thin 1.6.1 codename Death Proof
DEBUG: ---- End HTTP Status/Header Data ----
DEBUG: Loaded <Client chef-validator /Users/wilton/Documents/workspace/github/ironfan_homebase/knife/credentials/client_keys/client-chef-validator.pem>
DEBUG: Loaded <Client chef-webui /Users/wilton/Documents/workspace/github/ironfan_homebase/knife/credentials/client_keys/client-chef-webui.pem>
DEBUG: Loaded <Client wilton /Users/wilton/Documents/workspace/github/ironfan_homebase/knife/credentials/client_keys/client-wilton.pem>
DEBUG: Loaded <Client sandbox-simple-0 /Users/wilton/Documents/workspace/github/ironfan_homebase/knife/credentials/client_keys/client-sandbox-simple-0.pem>
sandbox: - loaded clients
sandbox: - loading machines
sandbox: - loading nodes
DEBUG: Signing the request as wilton
DEBUG: Sending HTTP Request via GET to chef-server.shrbt.com:4000//search/node
DEBUG: ---- HTTP Status and Header Data: ----
DEBUG: HTTP 1.1 200 OK
DEBUG: content-type: application/json; charset=utf-8
DEBUG: connection: close
DEBUG: server: thin 1.6.1 codename Death Proof
DEBUG: ---- End HTTP Status/Header Data ----
DEBUG: Loaded <Node sandbox-simple-0 role[systemwide], role[ssh], role[nfs_client], role[set_hostname], recipe[log_integration::logrotate], role[volumes], role[org_base], role[org_users], role[package_set], role[minidash], role[org_final], role[tuning], role[sandbox-cluster], role[sandbox-simple-facet]>
sandbox: - loaded nodes
sandbox: - loading roles
sandbox: - loading elastic_ips
DEBUG: Signing the request as wilton
DEBUG: Sending HTTP Request via GET to chef-server.shrbt.com:4000//search/role
sandbox: - loading ebs_volumes
sandbox: - loading keypairs
sandbox: - loading security_groups
DEBUG: ---- HTTP Status and Header Data: ----
DEBUG: HTTP 1.1 200 OK
DEBUG: content-type: application/json; charset=utf-8
DEBUG: connection: close
DEBUG: server: thin 1.6.1 codename Death Proof
DEBUG: ---- End HTTP Status/Header Data ----
DEBUG: Loaded <Role sandbox-cluster >
DEBUG: Loaded <Role sandbox-raid_demo-facet >
DEBUG: Loaded <Role sandbox-simple-facet >
sandbox: - loaded roles
sandbox: - loading iam_server_certificates
DEBUG: Loaded #<Ironfan::Provider::Ec2::ElasticIp(adaptee= <Fog::Compute::AWS::Address
public_ip="54.204.27.74",
allocation_id=nil,
server_id="i-c995cdb0",
network_interface_id=nil,
domain="standard"
>)>
DEBUG: AWS domain: standard
DEBUG: available ip match: 54.204.27.74
DEBUG: available allocation_id match:
DEBUG: ----------------------
sandbox: - loaded elastic_ips
DEBUG: Loaded <SecurityGroup sg-1100427a systemwide>
DEBUG: Loaded <SecurityGroup sg-1b004270 nfs_client>
DEBUG: Loaded <SecurityGroup sg-012eac6a default ["tcp:0-65535 (default | 38.121.238.132/32,74.90.167.4/32)", "udp:0-65535 (default | )", "icmp:-1--1 (default | )", "tcp:22-22 ( | 74.90.167.4/32)"]>
DEBUG: Loaded <SecurityGroup sg-7d2fad16 chef_client>
DEBUG: Loaded <SecurityGroup sg-13004278 ssh ["tcp:22-22 ( | 0.0.0.0/0)"]>
DEBUG: Loaded <SecurityGroup sg-1f004274 sandbox-raid_demo>
DEBUG: Loaded <SecurityGroup sg-632fad08 chef_server>
DEBUG: Loaded <SecurityGroup sg-1d004276 sandbox-simple>
DEBUG: Loaded <SecurityGroup sg-19004272 sandbox ["tcp:1-65535 (sandbox | )", "udp:1-65535 (sandbox | )", "icmp:-1--1 (sandbox | )"]>
sandbox: - loaded security_groups
DEBUG: Loaded <Keypair sandbox >
DEBUG: Loaded <Keypair dish_digital_engineering>
sandbox: - loaded keypairs
DEBUG: Loaded <Machine i-c995cdb0 2013-11-07 21:21:49 UTC chef-server 10.226.114.7 54.204.27.74 m1.xlarge us-east-1c dish_digital_engineering:chef_client,default,chef_server>
DEBUG: Loaded <Machine i-2f739c56 2013-12-19 22:33:19 UTC sandbox-simple-0 10.40.127.30 50.19.27.74 m1.large us-east-1c sandbox:sandbox,systemwide,ssh,sandbox-simple,nfs_client>
sandbox: - loaded machines
sandbox: - loaded iam_server_certificatesDEBUG: Loaded <EbsVolume vol-4df1910e 2013-11-07 21:21:52 UTC in-use /dev/sda1 8 i-c995cdb0:2013-11-07 21:21:52 UTC>
DEBUG: Loaded <EbsVolume vol-ba3d3af9 2013-12-19 22:33:22 UTC sandbox-simple-0-root in-use /dev/sda1 / 8 i-2f739c56:2013-12-19 22:33:22 UTC>
sandbox: - loaded ebs_volumes
sandbox: - loading elastic_load_balancers
sandbox: - loaded elastic_load_balancers
sandbox: Reconciling DSL and provider information
INFO: Inventoried 1 computers
DEBUG: Adding ec2-50-19-27-74.compute-1.amazonaws.com
DEBUG: establishing connection to ec2-50-19-27-74.compute-1.amazonaws.com:22
DEBUG: connection established
INFO: negotiating protocol version
DEBUG: remote is `SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.1'
DEBUG: local is `SSH-2.0-Ruby/Net::SSH_2.7.0 x86_64-darwin11.4.2'
DEBUG: read 984 bytes
DEBUG: received packet nr 0 type 20 len 980
INFO: got KEXINIT from server
INFO: sending KEXINIT
DEBUG: queueing packet nr 0 type 20 len 1620
DEBUG: sent 1624 bytes
INFO: negotiating algorithms
DEBUG: negotiated:
* kex: diffie-hellman-group-exchange-sha1
* host_key: ssh-rsa
* encryption_server: aes128-cbc
* encryption_client: aes128-cbc
* hmac_client: hmac-sha1
* hmac_server: hmac-sha1
* compression_client: none
* compression_server: none
* language_client:
* language_server:
DEBUG: exchanging keys
DEBUG: queueing packet nr 1 type 34 len 20
DEBUG: sent 24 bytes
DEBUG: read 152 bytes
DEBUG: received packet nr 1 type 31 len 148
DEBUG: queueing packet nr 2 type 32 len 140
DEBUG: sent 144 bytes
DEBUG: read 720 bytes
DEBUG: received packet nr 2 type 33 len 700
DEBUG: queueing packet nr 3 type 21 len 20
DEBUG: sent 24 bytes
DEBUG: received packet nr 3 type 21 len 12
DEBUG: beginning authentication of `ubuntu'
DEBUG: queueing packet nr 4 type 5 len 28
DEBUG: sent 52 bytes
DEBUG: read 52 bytes
DEBUG: received packet nr 4 type 6 len 28
DEBUG: trying none
DEBUG: Mechanism none was requested, but isn't a known type. Ignoring it.
DEBUG: trying publickey
DEBUG: connecting to ssh-agent
DEBUG: sending agent request 1 len 51
DEBUG: received agent packet 2 len 5
DEBUG: sending agent request 11 len 0
DEBUG: received agent packet 12 len 5
DEBUG: trying publickey (cd:5d:92:8c:ad:bd:6a:f6:2f:fe:7e:ae:21:60:72:43)
DEBUG: queueing packet nr 5 type 50 len 348
DEBUG: sent 372 bytes
DEBUG: read 52 bytes
DEBUG: received packet nr 5 type 51 len 28
DEBUG: allowed methods: publickey
ERROR: all authorization methods failed (tried none, publickey)
/usr/local/rvm/gems/ruby-1.9.3-p194@ironfan_homebase/gems/chef-10.30.2/lib/chef/knife/ssh.rb:100:in `block in session': undefined method `each' for nil:NilClass (NoMethodError)
from /usr/local/rvm/gems/ruby-1.9.3-p194@ironfan_homebase/gems/net-ssh-multi-1.1/lib/net/ssh/multi/session.rb:499:in `call'
from /usr/local/rvm/gems/ruby-1.9.3-p194@ironfan_homebase/gems/net-ssh-multi-1.1/lib/net/ssh/multi/session.rb:499:in `block in next_session'
from /usr/local/rvm/gems/ruby-1.9.3-p194@ironfan_homebase/gems/net-ssh-multi-1.1/lib/net/ssh/multi/session.rb:499:in `catch'
from /usr/local/rvm/gems/ruby-1.9.3-p194@ironfan_homebase/gems/net-ssh-multi-1.1/lib/net/ssh/multi/session.rb:499:in `rescue in next_session'
from /usr/local/rvm/gems/ruby-1.9.3-p194@ironfan_homebase/gems/net-ssh-multi-1.1/lib/net/ssh/multi/session.rb:482:in `next_session'
from /usr/local/rvm/gems/ruby-1.9.3-p194@ironfan_homebase/gems/net-ssh-multi-1.1/lib/net/ssh/multi/server.rb:138:in `session'
from /usr/local/rvm/gems/ruby-1.9.3-p194@ironfan_homebase/gems/net-ssh-multi-1.1/lib/net/ssh/multi/session_actions.rb:36:in `block (2 levels) in sessions'
kornypoet commented
@gwilton This is most likely a user issue. Run the cluster kick command as so and send the output if there is still an issue:
bundle exec knife cluster kick sandbox -x <ssh_username> -i /path/to/<ssh_username>/id_rsa
Where the <ssh_username> is a valid username on the sandbox cluster with ssh access using the private key you provide using the -i flag.
gwilton commented
@kornypoet Thanks for the reply. providing user and pem works. But just the user alone result with the same error. Can it be an issue finding the pem file?
[ironfan_homebase (master)]$ bundle exec knife cluster ssh sandbox uptime -x ubuntu -i knife/credentials/ec2_keys/sandbox.pem
Inventorying servers in sandbox cluster, all facets, all servers
sandbox: Loading chef
sandbox: Loading ec2
sandbox: Reconciling DSL and provider information
ec2-50-19-27-74.compute-1.amazonaws.com 14:11:28 up 15:37, 1 user, load average: 0.05, 0.03, 0.05
[ironfan_homebase (master)]$ bundle exec knife cluster ssh sandbox uptime -x ubuntu
Inventorying servers in sandbox cluster, all facets, all servers
sandbox: Loading chef
sandbox: Loading ec2
sandbox: Reconciling DSL and provider information
ERROR: knife encountered an unexpected error
This may be a bug in the 'cluster ssh' knife command or plugin
Please collect the output of this command with the `-VV` option before filing a bug report.
Exception: NoMethodError: undefined method `each' for nil:NilClass
joshbronson commented
This one isn't a user error. I took a look at the stacktrace the other day, and it looks like some recent changes I pushed in may have affected the way cluster ssh builds the targets that chef uses. Unfortunately, the cause wasn't immediately clear to me, and this is a busy time of the year for us, so I may not have a fix for this for a bit. Sorry if that keeps you from upgrading right now!
gwilton commented
Hey no problem, I will use the -x & -i options for now and continue testing things out. I will also dig a little deeper on this issue on my side. Happy holidays!
joshbronson commented
Thanks for the feed back!. fixed this in Ironfan. You can bump Ironfan to '~> 5.0.0' in ironfan-homebase, and everything should work fine. I should get a chance to test and bump that soon.