chef/knife-ec2

Knife EC2 Connect Not Using private_ip_address

Closed this issue · 3 comments

aschumpert commented

Version:

chef --version
Chef Workstation version: 0.17.5
Chef Infra Client version: 15.8.23
Chef InSpec version: 4.18.100
Chef CLI version: 2.0.0
Test Kitchen version: 2.4.0
Cookstyle version: 5.22.6

Environment:

  • OS: centos7
  • Cloud: AWS

Scenario:

Would like knife ec2 server create to use the attribute 'private_ip_address' when connecting to the newly created EC2 instance. Rather knife ssh is attempting to connect with the FQDN of the node. In my AWS environment, the FQDN is not resolvable, and knife fails to connect.

Steps to Reproduce:

  1. knife ec2 server create
knife ec2 server create \
  --connection-user ec2-user \
  --ssh-key me-key \
  --ssh-identity-file me-key.pem \
  --region us-east-1 \
  --image ami-000 \
  --flavor m5.xlarge \
  --ebs-size 60 \
  --subnet subnet-000 \
  --security-group-id sg-000 \
  --run-list '' -VV \
  --server-connect-attribute private_ip_address

Expected Result:

  1. Expecting knife to connect via the private_ip_address

Actual Result:

  1. knife is attempting to connect with the FQDN of the node. (exact IPs and a few other identifying details are scrubbed out)
INFO: Using configuration from /home/andrew/git/chef-repo/.chef/config.rb
DEBUG: Checking if we need to accept Chef license to bootstrap node
DEBUG: Reading products and relationships...
DEBUG: Successfully read products and relationships
DEBUG: License acceptance required for chef version: 15
DEBUG: Searching for the following licenses: ["infra-client", "inspec"]
DEBUG: Found license chef_infra_client at /home/andrew/.chef/accepted_licenses/chef_infra_client
DEBUG: Found license inspec at /home/andrew/.chef/accepted_licenses/inspec
DEBUG: Missing licenses remaining: []
DEBUG: All licenses present
DEBUG: Using AWS credential file at //home/andrew/git/chef-repo/.chef/aws_credentials
DEBUG: Using AWS profile default
DEBUG: Using AWS region us-east-1
DEBUG: Setting up AWS connection using aws_access_key_id: 000 aws_secret_access_key: 000 aws_session_token: 000

Waiting for EC2 to create the instance
Instance ID: i-000
Flavor: m5.xlarge
Image: ami-000
Region: us-east-1
Availability Zone: us-east-1a
Security Group Ids: sg-000
SSH Key: me-key
T2/T3 Unlimited: Disabled

Subnet ID: subnet-000
Tenancy: default
Private IP Address: x.x.x.94

Waiting for sshd access to become available
SSH Target Address: x.x.x.94(private_ip_address)
DEBUG: No ssh gateway found, making a direct connection
DEBUG: sshd accepting connections on x.x.x.94, banner is SSH-2.0-OpenSSH_7.4

done

SSH Target Address: x.x.x.94(private_ip_address)
Connecting to ip-x-x-x-94.us-east-1.compute.internal
DEBUG: [SSH] ec2-user@ip-x-x-x-94.us-east-1.compute.internal<{:user_known_hosts_file=>"/dev/null", :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>60, :auth_methods=>["none", "publickey"], :keys_only=>true, :keys=>["me-key.pem"], :password=>"<hidden>", :forward_agent=>nil, :non_interactive=>true, :verify_host_key=>:always}> (cmd.exe /c ver)
DEBUG: [SSH] opening connection to ec2-user@ip-x-x-x-94.us-east-1.compute.internal<{:user_known_hosts_file=>"/dev/null", :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>60, :auth_methods=>["none", "publickey"], :keys_only=>true, :keys=>["me-key.pem"], :password=>"<hidden>", :forward_agent=>nil, :non_interactive=>true, :verify_host_key=>:always}>
Traceback (most recent call last):
        39: from /bin/knife:362:in `<main>'
        38: from /bin/knife:362:in `load'
        37: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/bin/knife:24:in `<top (required)>'
        36: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/application/knife.rb:163:in `run'
        35: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/knife.rb:229:in `run'
        34: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/knife.rb:485:in `run_with_pretty_exceptions'
        33: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/local_mode.rb:42:in `with_server_connectivity'
        32: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/knife.rb:486:in `block in run_with_pretty_exceptions'
        31: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/knife/bootstrap.rb:574:in `run'
        30: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/knife/bootstrap.rb:625:in `connect!'
        29: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/knife/bootstrap.rb:697:in `do_connect'
        28: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/knife/bootstrap/train_connector.rb:70:in `connect!'
        27: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/chef-15.8.23/lib/chef/knife/bootstrap/train_connector.rb:57:in `connection'
        26: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/transports/ssh.rb:82:in `connection'
        25: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/transports/ssh.rb:240:in `create_new_connection'
        24: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/transports/ssh.rb:240:in `new'
        23: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/transports/ssh_connection.rb:53:in `initialize'
        22: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/extras/command_wrapper.rb:166:in `load'
        21: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/plugins/base_connection.rb:116:in `platform'
        20: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect.rb:9:in `scan'
        19: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/scanner.rb:27:in `scan'
        18: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/scanner.rb:27:in `each'
        17: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/scanner.rb:33:in `block in scan'
        16: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/scanner.rb:45:in `scan_children'
        15: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/scanner.rb:45:in `each'
        14: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/scanner.rb:46:in `block in scan_children'
        13: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/scanner.rb:46:in `instance_eval'
        12: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/specifications/os.rb:29:in `block in load'
        11: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/helpers/os_windows.rb:4:in `detect_windows'
        10: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/platforms/detect/helpers/os_windows.rb:9:in `check_cmd'
         9: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/plugins/base_connection.rb:128:in `run_command'
         8: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/transports/ssh_connection.rb:223:in `run_command_via_connection'
         7: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/transports/ssh_connection.rb:254:in `session'
         6: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/train-core-3.2.0/lib/train/transports/ssh_connection.rb:185:in `establish_connection'
         5: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:246:in `start'
         4: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:246:in `new'
         3: from /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:73:in `initialize'
         2: from /opt/chef-workstation/embedded/lib/ruby/2.6.0/socket.rb:631:in `tcp'
         1: from /opt/chef-workstation/embedded/lib/ruby/2.6.0/socket.rb:227:in `foreach'
/opt/chef-workstation/embedded/lib/ruby/2.6.0/socket.rb:227:in `getaddrinfo': getaddrinfo: Name or service not known (SocketError)
rkeiii commented

I am running on an Ubuntu 18.04 workstation using Chef DK 4.7.73 and I'm seeing this issue as well.

Chef DK version info

rkeiii@ate:~$ dpkg -l | grep -i chef
ii  chefdk                                     4.7.73-1                                        amd64        The full stack of chefdk
rkeiii@ate:~$ which gem
/opt/chefdk/embedded/bin/gem
rkeiii@ate:~$ gem info knife-ec2

*** LOCAL GEMS ***

knife-ec2 (1.0.28)
    Author: Chef Software, Inc.
    Homepage: https://github.com/chef/knife-ec2
    License: Apache-2.0
    Installed at: /opt/chefdk/embedded/lib/ruby/gems/2.6.0

    Amazon EC2 Support for Chef's Knife Command
rkeiii@ate:~$

Example of 'knife ec2 server create -VVV ...' not respecting the --service-connect-attribute param

rkeiii@ate:~/workspace/chef-repo/scripts$ knife ec2 server create -VVV -I ami-04ac550b78324f651 -S test-key --subnet subnet-f11eae9f --associate-eip 52.73.55.126 -g sg-57cb662b -g sg-e8238687 -g sg-e94df086 -g sg-da3aedb5 -N search2.qa -f r5.xlarge -E qa --run-list 'role[xenial_search_class]' --bootstrap-template ../bootstrap/chef-full.erb --iam-profile test-ec2-role --connection-user ubuntu --ssh-forward-agent --server-connect-attribute private_ip_address
INFO: Using configuration from /home/rkeiii/.chef/config.rb
DEBUG: Checking if we need to accept Chef license to bootstrap node
DEBUG: Reading products and relationships...
DEBUG: Successfully read products and relationships
DEBUG: Using AWS credential file at /home/rkeiii/.aws/credentials
DEBUG: Using AWS profile profile test-user-mfa
DEBUG: Using AWS region us-east-1
DEBUG: Setting up AWS connection using aws_access_key_id: ASIAXXXXXXXXXXXXXXXX aws_secret_access_key: PH+sXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX aws_session_token: FwoGXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Waiting for EC2 to create the instance
Instance ID: i-343433c9dfa38d202
Flavor: r5.xlarge
Image: ami-04ac550b78324f651
Region: us-east-1
Availability Zone: us-east-1d
Security Group Ids: sg-e94df086, sg-e8238687, sg-57cb662b, sg-da3aedb5
IAM Profile: test-ec2-role
AWS Tags: Name: search2.qa
SSH Key: test-key
T2/T3 Unlimited: Disabled

Subnet ID: subnet-f11eae9f
Tenancy: default
Public IP Address: 52.73.55.126
Private IP Address: 10.20.151.169

Waiting for sshd access to become available
SSH Target Address: 10.20.151.169(private_ip_address)
DEBUG: No ssh gateway found, making a direct connection
DEBUG: ssh failed to connect: 10.20.151.169
.DEBUG: ssh failed to connect: 10.20.151.169
.DEBUG: sshd accepting connections on 10.20.151.169, banner is SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8

done

SSH Target Address: 10.20.151.169(private_ip_address)
Connecting to ec2-52-73-55-126.compute-1.amazonaws.com
DEBUG: [SSH] Using Agent keys as no password or key file have been specified
DEBUG: [SSH] ubuntu@ec2-52-73-55-126.compute-1.amazonaws.com<{:user_known_hosts_file=>"/dev/null", :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>60, :auth_methods=>["none", "publickey"], :keys_only=>nil, :keys=>[], :password=>"<hidden>", :forward_agent=>nil, :non_interactive=>true, :verify_host_key=>:always}> (cmd.exe /c ver)
breisig commented

I am having the EXACT same issue. It's related to the partial fix in issue #633 [ https://github.com//issues/633.

Using the Private DNS name causes workstations to not be able to connect if they are not running within the VPC. In our case , my Chef Workstation is on my local workstation in our office network with VPN Tunnel connectivity to the VPC and I cannot resolve the Internal AWS Private DNS name and ssh fails.

I don't really understand what use case would require the ssh connection via DNS, we've always used the IP in the past.

This fixes it for me.

/lib/chef/knife/ec2_server_create.rb

     def server_name
        return nil unless server

        if !server.public_dns_name.empty?
          server.public_dns_name
        #elsif !server.private_dns_name.empty?
        #    server.private_dns_name
        else
          server.private_ip_address
        end
      end

If we could remove the private_dns_name part from the code, that would fix it.

aschumpert commented

I am having the EXACT same issue. It's related to the partial fix in issue #633 [ https://github.com//issues/633.

Using the Private DNS name causes workstations to not be able to connect if they are not running within the VPC. In our case , my Chef Workstation is on my local workstation in our office network with VPN Tunnel connectivity to the VPC and I cannot resolve the Internal AWS Private DNS name and ssh fails.

I don't really understand what use case would require the ssh connection via DNS, we've always used the IP in the past.

This fixes it for me.

/lib/chef/knife/ec2_server_create.rb

     def server_name
        return nil unless server

        if !server.public_dns_name.empty?
          server.public_dns_name
        #elsif !server.private_dns_name.empty?
        #    server.private_dns_name
        else
          server.private_ip_address
        end
      end

If we could remove the private_dns_name part from the code, that would fix it.

I've confirmed this workaround is suitable for my scenario as well in Chef-workstation
/opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/knife-ec2-1.0.32/lib/chef/knife/ec2_server_create.rb