Timeout on Docker -- ssh_exchange_identification: Connection closed by remote host
c10l opened this issue · 26 comments
Beaker is failing on CentOS 6, running on Docker 1.4.1 from epel-testing. It starts the container, but then is unable to ssh in to it.
Beaker verbose output:
-bash-4.1$ bundle exec rspec spec/acceptance/
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/opt/go/agent0/.gems/ruby/2.1.0/gems/specinfra-2.12.1/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here
Hypervisor for centos6 is docker
Beaker::Hypervisor, found some docker boxes to create
Provisioning docker
provisioning centos6
Creating image
Dockerfile is FROM chef/centos-6
RUN yum clean all
RUN yum install -y sudo openssh-server openssh-clients curl ntpdate
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN echo root:root | chpasswd
RUN mkdir -p /root/.ssh
RUN yum -y install tar
EXPOSE 22
CMD ["/sbin/init"]
post
/v1.15/build
{:rm=>true}
Dockerfile0000640000000000000000000000064412462207066013315 0ustar00wheelwheel00000000000000 FROM chef/centos-6
RUN yum clean all
RUN yum install -y sudo openssh-server openssh-clients curl ntpdate
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN echo root:root | chpasswd
RUN mkdir -p /root/.ssh
RUN yum -y install tar
EXPOSE 22
CMD ["/sbin/init"]
Creating container from image 99a2b5945d5f
post
/v1.15/containers/create
{}
{"Image":"99a2b5945d5f","Hostname":"centos6"}
Starting container b11e44daf226c7044e4a8220e84bbc530acc1e875fc74a5e38772e4106af32dc
post
/v1.15/containers/b11e44daf226c7044e4a8220e84bbc530acc1e875fc74a5e38772e4106af32dc/start
{}
{"PublishAllPorts":true,"Privileged":true}
get
/v1.15/containers/b11e44daf226c7044e4a8220e84bbc530acc1e875fc74a5e38772e4106af32dc/json
{}
get
/v1.15/containers/b11e44daf226c7044e4a8220e84bbc530acc1e875fc74a5e38772e4106af32dc/json
{}
node available as ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@0.0.0.0 -p 49155
centos6 16:28:38$ rpm -q curl
Warning: Try 1 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 1 seconds
Warning: Try 2 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 1 seconds
Warning: Try 3 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 2 seconds
Warning: Try 4 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 3 seconds
Warning: Try 5 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 5 seconds
Warning: Try 6 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 8 seconds
Warning: Try 7 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 13 seconds
Warning: Try 8 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 21 seconds
Warning: Try 9 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 34 seconds
Warning: Try 10 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 55 seconds
Warning: Try 11 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 89 seconds
Failed to connect to 0.0.0.0
Failed: errored in validate
#<Net::SSH::Disconnect: connection closed by remote host>
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:50
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:46
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:45
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:45
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:43
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:43
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:32
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84
/opt/rubies/ruby-2.1.3/lib/ruby/2.1.0/timeout.rb:76
/opt/rubies/ruby-2.1.3/lib/ruby/2.1.0/timeout.rb:127
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh.rb:207
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh.rb:207
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/ssh_connection.rb:42
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/ssh_connection.rb:33
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host.rb:242
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host.rb:270
/opt/rubies/ruby-2.1.3/lib/ruby/2.1.0/benchmark.rb:294
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host.rb:269
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host/unix/pkg.rb:28
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:108
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:107
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:107
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:95
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/shared/host_manager.rb:74
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/shared/host_manager.rb:71
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/shared/host_manager.rb:70
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/shared/host_manager.rb:70
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/dsl/patterns.rb:32
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:84
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/hypervisor.rb:117
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/network_manager.rb:61
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/network_manager.rb:60
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/network_manager.rb:60
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec/beaker_shim.rb:36
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec/spec_helper.rb:43
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core.rb:81
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec/spec_helper.rb:5
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:5
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:5
/opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:1
/opt/go/agent0/.gems/ruby/2.1.0/bundler/gems/gamesys_puppet_spec_helper-56066fb0e638/lib/gamesys_puppet_spec_helper/spec_helper_acceptance.rb:1
/opt/go/agent0/.gems/ruby/2.1.0/bundler/gems/gamesys_puppet_spec_helper-56066fb0e638/lib/gamesys_puppet_spec_helper/spec_helper_acceptance.rb:1
/opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/spec_helper_acceptance.rb:2
/opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/spec_helper_acceptance.rb:2
/opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/acceptance/init_spec.rb:1
/opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/acceptance/init_spec.rb:1
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:96
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:84
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:69
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:37
/opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/exe/rspec:4
/opt/go/agent0/.gems/ruby/2.1.0/bin/rspec:23
/opt/go/agent0/.gems/ruby/2.1.0/bin/rspec:23
/opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:50:in `rescue in block (2 levels) in negotiate!': connection closed by remote host (Net::SSH::Disconnect)
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:46:in `block (2 levels) in negotiate!'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:45:in `loop'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:45:in `block in negotiate!'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:43:in `loop'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:43:in `negotiate!'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:32:in `initialize'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `new'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `block in initialize'
from /opt/rubies/ruby-2.1.3/lib/ruby/2.1.0/timeout.rb:76:in `timeout'
from /opt/rubies/ruby-2.1.3/lib/ruby/2.1.0/timeout.rb:127:in `timeout'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `initialize'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh.rb:207:in `new'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh.rb:207:in `start'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/ssh_connection.rb:42:in `connect'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/ssh_connection.rb:33:in `connect'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host.rb:242:in `connection'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host.rb:270:in `block in exec'
from /opt/rubies/ruby-2.1.3/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host.rb:269:in `exec'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host/unix/pkg.rb:28:in `check_for_package'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:108:in `block in check_and_install_packages_if_needed'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:107:in `each'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:107:in `check_and_install_packages_if_needed'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:95:in `block in validate_host'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/shared/host_manager.rb:74:in `run_block_on'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/shared/host_manager.rb:71:in `block in run_block_on'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/shared/host_manager.rb:70:in `map'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/shared/host_manager.rb:70:in `run_block_on'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/dsl/patterns.rb:32:in `block_on'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/host_prebuilt_steps.rb:84:in `validate_host'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/hypervisor.rb:117:in `validate'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/network_manager.rb:61:in `block in validate'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/network_manager.rb:60:in `each_key'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-2.0.0/lib/beaker/network_manager.rb:60:in `validate'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec/beaker_shim.rb:36:in `validate'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec/spec_helper.rb:43:in `block in <top (required)>'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core.rb:81:in `configure'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec/spec_helper.rb:5:in `<top (required)>'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:5:in `require'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:5:in `<module:BeakerRSpec>'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:1:in `<top (required)>'
from /opt/go/agent0/.gems/ruby/2.1.0/bundler/gems/gamesys_puppet_spec_helper-56066fb0e638/lib/gamesys_puppet_spec_helper/spec_helper_acceptance.rb:1:in `require'
from /opt/go/agent0/.gems/ruby/2.1.0/bundler/gems/gamesys_puppet_spec_helper-56066fb0e638/lib/gamesys_puppet_spec_helper/spec_helper_acceptance.rb:1:in `<top (required)>'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/spec_helper_acceptance.rb:2:in `require'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/spec_helper_acceptance.rb:2:in `<top (required)>'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/acceptance/init_spec.rb:1:in `require'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/acceptance/init_spec.rb:1:in `<top (required)>'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `load'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `block in load_spec_files'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `each'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `load_spec_files'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:96:in `setup'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:84:in `run'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:69:in `run'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:37:in `invoke'
from /opt/go/agent0/.gems/ruby/2.1.0/gems/rspec-core-3.1.7/exe/rspec:4:in `<top (required)>'
from /opt/go/agent0/.gems/ruby/2.1.0/bin/rspec:23:in `load'
from /opt/go/agent0/.gems/ruby/2.1.0/bin/rspec:23:in `<main>'
I tried the SSH command manually on the host and got this:
-bash-4.1$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@0.0.0.0 -p 49155
ssh_exchange_identification: Connection closed by remote host
This is my nodeset file:
---
HOSTS:
centos6:
roles:
- master
platform: el-6-x86_64
image: chef/centos-6
hypervisor: docker
docker_image_commands:
- 'mkdir -p /root/.ssh'
- 'yum -y install tar'
docker_cmd: '["/sbin/init"]'
docker_preserve_image: false
CONFIG:
type: foss
log_level: debug
Since you do a mkdir of the .ssh dir it could fail on incorrect rights on the .ssh dir.
if you use beaker 2.2.0 it should be solved in there and you can remove the mkdir command as well.
I'm experiencing the same issue on a CentOS-6 host.. but on a CentOS-7 host, with the exact same SUT yaml configs, the docker runs work perfectly fine. I'm pretty sure Beaker/Docker ran fine on Centos6 until fairly recently, but I've not run the tests since upgrading to 2.2.0
Interestingly, if I remove docker_cmd
completely, it defaults to /usr/sbin/ssh -D
, and it connects okay until Beaker attempts a service sshd restart
, at which point the daemon disconnects and the test fails.
I also assumed it would be caused by the mkdir /root/.ssh
I'd put in place to address that missing, but it also fails without that image command being present
It fails on docker-io-1.3.2-2.el6.x86_64
from epel, as well as docker-io-1.4.1-3.el6.x86_64
from epel-testing
nodeset:
---
HOSTS:
centos6:
roles:
- master
platform: el-6-x86_64
image: centos:centos6
hypervisor: docker
docker_cmd: '["/sbin/init"]'
docker_image_commands:
- 'yum install -y tar gcc ruby-devel git'
CONFIG:
type: foss
log_level: verbose
forward_ssh_agent: true
Same here. With /sbin/init
as the docker_cmd
it times out same as before. Without it, the connection is lost when sshd
restarts.
-bash-4.1$ bundle exec rake beaker
/opt/rubies/ruby-2.1.3/bin/ruby -I/opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib:/opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-support-3.1.2/lib /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/exe/rspec spec/acceptance --color
/opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/opt/go/agent0/.gem/ruby/2.1.3/gems/specinfra-2.12.1/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here
Hypervisor for centos6 is docker
Beaker::Hypervisor, found some docker boxes to create
Provisioning docker
provisioning centos6
Creating image
Dockerfile is FROM chef/centos-6
RUN yum clean all
RUN yum install -y sudo openssh-server openssh-clients curl ntpdate
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN echo root:root | chpasswd
RUN yum -y install tar
EXPOSE 22
CMD /usr/sbin/sshd -D
post
/v1.15/build
{:rm=>true}
Dockerfile0000640000000000000000000000062012462403236013304 0ustar00wheelwheel00000000000000 FROM chef/centos-6
RUN yum clean all
RUN yum install -y sudo openssh-server openssh-clients curl ntpdate
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN echo root:root | chpasswd
RUN yum -y install tar
EXPOSE 22
CMD /usr/sbin/sshd -D
Creating container from image df99d6c15b5c
post
/v1.15/containers/create
{}
{"Image":"df99d6c15b5c","Hostname":"centos6"}
Starting container 3f6916ee03df033848d784e2a101113410a9498a8e9fbe6adf374afcc8841ad7
post
/v1.15/containers/3f6916ee03df033848d784e2a101113410a9498a8e9fbe6adf374afcc8841ad7/start
{}
{"PublishAllPorts":true,"Privileged":true}
get
/v1.15/containers/3f6916ee03df033848d784e2a101113410a9498a8e9fbe6adf374afcc8841ad7/json
{}
get
/v1.15/containers/3f6916ee03df033848d784e2a101113410a9498a8e9fbe6adf374afcc8841ad7/json
{}
node available as ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@0.0.0.0 -p 49159
centos6 10:08:41$ rpm -q curl
Warning: Try 1 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 1 seconds
Created ssh connection to 0.0.0.0, user: root, opts: {:password=>"root", :port=>"49159", :forward_agent=>false, :user=>"root"}
curl-7.19.7-40.el6_6.3.x86_64
centos6 executed in 1.22 seconds
centos6 10:08:42$ rpm -q ntpdate
ntpdate-4.2.6p5-2.el6.centos.x86_64
centos6 executed in 0.03 seconds
centos6 10:08:43$ echo "/usr/bin"
/usr/bin
centos6 executed in 0.01 seconds
centos6 10:08:43$ echo "/opt/puppet-git-repos/hiera/bin"
/opt/puppet-git-repos/hiera/bin
centos6 executed in 0.01 seconds
centos6 10:08:43$ echo "/opt/puppet-git-repos/hiera/lib"
/opt/puppet-git-repos/hiera/lib
centos6 executed in 0.01 seconds
centos6 10:08:43$ echo "/opt/puppet-git-repos/hiera-puppet/lib"
/opt/puppet-git-repos/hiera-puppet/lib
centos6 executed in 0.01 seconds
setting local environment on centos6
centos6 10:08:43$ echo '
PermitUserEnvironment yes' >> /etc/ssh/sshd_config
centos6 executed in 0.01 seconds
centos6 10:08:43$ /sbin/service sshd restart
Stopping sshd: /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/ruby_compat.rb:30:in `select': closed stream (IOError)
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/ruby_compat.rb:30:in `io_select'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/packet_stream.rb:75:in `available_for_read?'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/packet_stream.rb:87:in `next_packet'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:183:in `block in poll_message'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:178:in `loop'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:178:in `poll_message'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:461:in `dispatch_incoming_packets'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:222:in `preprocess'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:206:in `process'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170:in `block in loop'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170:in `loop'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170:in `loop'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/ssh_connection.rb:96:in `try_to_execute'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/ssh_connection.rb:107:in `execute'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/host.rb:271:in `block in exec'
from /opt/rubies/ruby-2.1.3/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/host.rb:270:in `exec'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/host_prebuilt_steps.rb:489:in `block in set_env'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/shared/host_manager.rb:74:in `run_block_on'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/shared/host_manager.rb:71:in `block in run_block_on'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/shared/host_manager.rb:70:in `map'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/shared/host_manager.rb:70:in `run_block_on'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/dsl/patterns.rb:32:in `block_on'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/host_prebuilt_steps.rb:471:in `set_env'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/hypervisor.rb:111:in `configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/network_manager.rb:71:in `block in configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/network_manager.rb:70:in `each_key'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/network_manager.rb:70:in `configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec/beaker_shim.rb:43:in `configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec/spec_helper.rb:44:in `block in <top (required)>'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core.rb:81:in `configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec/spec_helper.rb:5:in `<top (required)>'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:5:in `require'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:5:in `<module:BeakerRSpec>'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:1:in `<top (required)>'
from /opt/go/agent0/.gem/ruby/2.1.3/bundler/gems/gamesys_puppet_spec_helper-ddcaf8a93a98/lib/gamesys_puppet_spec_helper/spec_helper_acceptance.rb:1:in `require'
from /opt/go/agent0/.gem/ruby/2.1.3/bundler/gems/gamesys_puppet_spec_helper-ddcaf8a93a98/lib/gamesys_puppet_spec_helper/spec_helper_acceptance.rb:1:in `<top (required)>'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/spec_helper_acceptance.rb:2:in `require'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/spec_helper_acceptance.rb:2:in `<top (required)>'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/acceptance/init_spec.rb:1:in `require'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/acceptance/init_spec.rb:1:in `<top (required)>'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `load'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `block in load_spec_files'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `each'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `load_spec_files'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:96:in `setup'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:84:in `run'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:69:in `run'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:37:in `invoke'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/exe/rspec:4:in `<main>'
/opt/rubies/ruby-2.1.3/bin/ruby -I/opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib:/opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-support-3.1.2/lib /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/exe/rspec spec/acceptance --color failed
Could you do a quick test with docker image electrical/centos:6.4-1
instead of the one you are using now?
Just to check if the docker image is misbehaving.
Same behaviour -- timeout with docker_cmd: '["/sbin/init"]'
, disconnection upon sshd restart
without it.
Anything else can provide?
Very weird. because i do the same and has always been working fine for me.
https://github.com/elasticsearch/puppet-elasticsearch/blob/master/spec/acceptance/nodesets/centos-6-x64.yml
One major difference I've noticed, is that this only fails on a centos 6.6 host for me (with docker installed via EPEL), the exact same nodeset works fine on a centos 7 host (with docker installed via the standard centos 7 repos) - I've not tested on any other operating systems though.
I'm running Docker 1.4.1 installed from epel-testing on a CentOS 6.6 host. Can't test it on CentOS 7 as we run stuff on xen and no one has been able to make that combination work yet around here. :)
I'm running on a Ubuntu 14.04 host with docker 1.4.1
How can I investigate this issue further? I'm at a complete loss here...
Must admit its very confusing. Could you try out my exact node config? wonder if that works. it it doesn't i'm fairly out of idea's.
Same results...
It looks as though sshd is not being started in the container when run on a centos6 host; when I attach to a running container that Beaker's created with docker_cmd: '["/sbin/init"]'
, I don't seem to see a running ssh daemon, as far as I can tell:
-bash-4.1$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fc4f926000d0 0364c8063090 "/sbin/init" About a minute ago Up About a minute 0.0.0.0:49153->22/tcp nostalgic_lalande
-bash-4.1$ docker exec -i -t fc4f926000d0 bash
[root@centos6 /]# ps auxff
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 85 0.0 0.0 14724 1804 ? S 17:48 0:00 bash
root 99 0.0 0.0 16624 1044 ? R+ 17:48 0:00 \_ ps auxff
root 1 0.0 0.0 17128 1244 ? Ss 17:46 0:00 /sbin/init
root 11 0.0 0.0 14860 1908 ? Ss 17:46 0:00 /bin/bash /etc/rc.d/rc.sysinit
root 28 0.0 0.0 14972 1904 ? S 17:46 0:00 \_ /bin/bash /sbin/start_udev
root 66 0.2 0.0 8416 720 ? S 17:46 0:00 \_ /sbin/udevadm settle
root 60 0.0 0.0 10640 492 ? S<s 17:46 0:00 /sbin/udevd -d
By contrast, on the centos7 host, when I attach to the container from the same nodeset YAML:
[root@ip-172-31-12-185 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
65bca78c5cc6 a1a7ac8946dc "/sbin/init" 4 seconds ago Up 3 seconds 0.0.0.0:49590->22/tcp jovial_mestorf
91515fc1153f a1a7ac8946dc "/sbin/init" 28 hours ago Up 28 hours 0.0.0.0:49185->22/tcp romantic_wilson
[root@ip-172-31-12-185 ~]# docker exec -i -t 65bca78c5cc6 bash
[root@centos6 /]# ps auxff
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 679 0.4 0.0 108348 1680 pts/2 S 18:33 0:00 bash
root 799 0.0 0.0 110200 1056 pts/2 R+ 18:33 0:00 \_ ps auxff
root 1 0.1 0.0 19284 1500 ? Ss 18:33 0:00 /sbin/init
root 67 0.4 0.0 10824 908 ? S<s 18:33 0:00 /sbin/udevd -d
root 474 0.0 0.0 4132 652 ? Ss 18:33 0:00 /sbin/agetty /dev/ttyS0 115200 vt100-nav
root 476 0.0 0.0 4116 596 tty1 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty1
root 478 0.0 0.0 4116 596 tty2 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty2
root 480 0.0 0.0 4116 596 tty3 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty3
root 482 0.0 0.0 4116 596 tty4 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty4
root 484 0.0 0.0 4116 592 tty5 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty5
root 486 0.0 0.0 4116 592 tty6 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty6
root 516 0.0 0.0 66740 1176 ? Ss 18:33 0:00 /usr/sbin/sshd
root 533 0.5 0.0 96516 4476 ? Ss 18:33 0:00 \_ sshd: root@notty
root 724 16.3 0.2 142756 40024 ? Ssl 18:33 0:00 \_ /usr/bin/ruby /usr/bin/puppet apply --verbose --detailed-exitcodes /tmp/apply_manifest.pp.7kNnS6
[root@centos6 /]#
So, it appears that sshd is not starting correctly when run on a centos6 host, afaict ..
exactly why that's happening, is beyond me, I'm afraid..
Very weird.. could it be a kernel issue? which kernel is it on centos 6 and 7 ?
I suspect that it could be the kernel - there are mentions of issues with 2.6.x kernels in the docker github issue tracker - but afaict, they're all either closed or resolved .. so unless it's a very recent development, it's possible that not many others are experiencing the same issue. In addition to this, I'm not sure whether this is specific to Beaker (or the docker-api gem) or whether it's a general issue with docker on centos6 (due to limitations with the 2.6.x kernels)
What's very confusing is that I could have sworn it was working on centos6 previously, I'm just not sure when it stopped working. I'm going to try with an earlier version of Beaker at some point, but I won't be able to get to this in a while... @cassianoleal - is that something you could try: earlier releases of Beaker, as well as the docker-api gem?
-bash-4.1$ uname -a
Linux puppetci-agent0 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
AFAIK all Docker incompatibilities have been solved on this kernel (it's the latest on CentOS 6).
I'll try to downgrade the gems as @madAndroid suggested and see what happens.
For the record, I spawned a container using the Dockerfile that Beaker generates, then attached to it and sshd
is definitely running:
-bash-4.1$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ed4c51ff27c0 dfb7dc321cea "/sbin/init" 4 seconds ago Up 3 seconds 22/tcp determined_archimedes
-bash-4.1$ cat Dockerfile
FROM electrical/centos:6.4
RUN yum clean all
RUN yum install -y sudo openssh-server openssh-clients curl ntpdate
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN echo root:root | chpasswd
RUN yum install -y wget ntpdate rubygems ruby-augeas ruby-devel augeas-devel
RUN touch /etc/sysconfig/network
EXPOSE 22
CMD ["/sbin/init"]
-bash-4.1$ docker exec -i -t ed4 /bin/bash
bash-4.1# ps auxff
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 460 0.2 0.0 11304 1556 ? S 08:20 0:00 /bin/bash
root 468 0.0 0.0 13368 1000 ? R+ 08:20 0:00 \_ ps auxff
root 1 0.0 0.0 19232 1536 ? Ss 08:03 0:00 /sbin/init
root 78 0.0 0.0 10640 624 ? Ss 08:03 0:00 /sbin/udevd -d
root 242 0.0 0.0 66688 1228 ? Ss 08:03 0:00 /usr/sbin/sshd
root 467 0.0 0.0 4060 672 ? Ss 08:20 0:00 /sbin/mingetty /dev/tty[1-6]
Tried it with Beaker 1.20.1 and hit the same timeout.
While Beaker was trying to SSH into the container, I attached to it to run ps
and noticed that sshd
was indeed not running, so I started it:
bash-4.1# service sshd start
Generating SSH1 RSA host key: [ OK ]
Starting sshd: [ OK ]
And then Beaker was able to connect, but what called my attention was the first line where it says it's generating the RSA host key. From what I can tell, that's done as part of the image building process, right?
I've also tried it with docker-api
1.15.0 and saw the same results: no sshd
running...
Most likely because the older dockerapi gems have an API miss match with the docker api it self?
How does that explain that I have the same results with 1.15.0 and 1.17.0 (which is the latest)?
ah sorry. i miss understood your findings.
Must admit from my side i never ran on 2.6.x kernels for docker.. always been on 3.x ( currently on 3.13.x )
That does seem like the cleanest option, glad to hear you've made some
progress :)
On Mon, Feb 2, 2015 at 4:30 PM, cassiano notifications@github.com wrote:
This PR #626 #626 solves my
issue.—
Reply to this email directly or view it on GitHub
#652 (comment).
I'll close this out when the PR626 gets merged.
PR merged, closing.