Ubuntu AutoScalingGroup install.sh: Exited with error code 1
atamgp opened this issue · 6 comments
Metadata:
- Operating System: Ubuntu 16.04.4 LTS
- Installation method: install.sh script
- AWS CLI Version: aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-1061-aws botocore/1.4.70
I am using this in combination with an autoscale group of 1 (min and max). Normal EC2 used to work, but shifting to autoscale gives an error: Exited with error code 1
It seems that it gets stuck on the last command (./install_restart_sshd.sh ???) in the install.sh script, see log below.
When I log into the instance, and do:
"sudo install.sh ' it goes ok.
But calling
/usr/local/bin/cfn-init --verbose --stack XXX--region eu-west-1 --resource BastionLaunchConfig
goed bad.
UserData: |
# trap '/usr/local/bin/cfn-signal -e 1 --stack ${STACKNAME} --region ${REGION} --resource BastionAutoScalingGroup' ERR
apt-get update
apt-get -y install python-pip
pip install --upgrade pip
pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
# apt-get -y install python-setuptools
# mkdir aws-cfn-bootstrap-latest
# curl -s -m 60 https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1
# easy_install aws-cfn-bootstrap-latest
/usr/local/bin/cfn-signal -e 0 --stack ${STACKNAME} --region ${REGION} --resource BastionAutoScalingGroup #$?
/usr/local/bin/cfn-init --verbose --stack ${STACKNAME} --region ${REGION} --resource BastionLaunchConfig
Log:
Command b_install
2018-08-06 09:03:50,085 P8293 [INFO] -----------------------Command Output-----------------------
2018-08-06 09:03:50,085 P8293 [INFO] + getopts :hva:i:l:s:p:u:r: opt
2018-08-06 09:03:50,085 P8293 [INFO] + case $opt in
2018-08-06 09:03:50,085 P8293 [INFO] + IAM_GROUPS=i2d-dev-redshift-access-group
2018-08-06 09:03:50,086 P8293 [INFO] + getopts :hva:i:l:s:p:u:r: opt
2018-08-06 09:03:50,086 P8293 [INFO] + export IAM_GROUPS
2018-08-06 09:03:50,086 P8293 [INFO] + export SUDO_GROUPS
2018-08-06 09:03:50,086 P8293 [INFO] + export LOCAL_GROUPS
2018-08-06 09:03:50,086 P8293 [INFO] + export ASSUME_ROLE
2018-08-06 09:03:50,086 P8293 [INFO] + export USERADD_PROGRAM
2018-08-06 09:03:50,086 P8293 [INFO] + export USERADD_ARGS
2018-08-06 09:03:50,086 P8293 [INFO] ++ which aws
2018-08-06 09:03:50,086 P8293 [INFO] + '[' -x /usr/bin/aws ']'
2018-08-06 09:03:50,086 P8293 [INFO] ++ mktemp -d
2018-08-06 09:03:50,086 P8293 [INFO] + tmpdir=/tmp/tmp.NG3V1Pox2x
2018-08-06 09:03:50,086 P8293 [INFO] + cd /tmp/tmp.NG3V1Pox2x
2018-08-06 09:03:50,086 P8293 [INFO] + git clone -b master https://github.com/widdix/aws-ec2-ssh.git
2018-08-06 09:03:50,086 P8293 [INFO] Cloning into 'aws-ec2-ssh'...
2018-08-06 09:03:50,086 P8293 [INFO] + cd /tmp/tmp.NG3V1Pox2x/aws-ec2-ssh
2018-08-06 09:03:50,087 P8293 [INFO] + cp authorized_keys_command.sh /opt/authorized_keys_command.sh
2018-08-06 09:03:50,087 P8293 [INFO] + cp import_users.sh /opt/import_users.sh
2018-08-06 09:03:50,087 P8293 [INFO] + '[' i2d-dev-redshift-access-group '!=' '' ']'
2018-08-06 09:03:50,087 P8293 [INFO] + echo 'IAM_AUTHORIZED_GROUPS="i2d-dev-redshift-access-group"'
2018-08-06 09:03:50,087 P8293 [INFO] + '[' '' '!=' '' ']'
2018-08-06 09:03:50,087 P8293 [INFO] + '[' '' '!=' '' ']'
2018-08-06 09:03:50,087 P8293 [INFO] + '[' '' '!=' '' ']'
2018-08-06 09:03:50,087 P8293 [INFO] + '[' '' '!=' '' ']'
2018-08-06 09:03:50,087 P8293 [INFO] + '[' '' '!=' '' ']'
2018-08-06 09:03:50,087 P8293 [INFO] + ./install_configure_selinux.sh
2018-08-06 09:03:50,087 P8293 [INFO] + ./install_configure_sshd.sh
2018-08-06 09:03:50,087 P8293 [INFO] + cat
2018-08-06 09:03:50,087 P8293 [INFO] + chmod 0644 /etc/cron.d/import_users
2018-08-06 09:03:50,087 P8293 [INFO] + /opt/import_users.sh
2018-08-06 09:03:50,088 P8293 [INFO] ------------------------------------------------------------
2018-08-06 09:03:50,088 P8293 [ERROR] Exited with error code 1
I have some more information.
When I use Sub in the command of cloudf init, the command parses ok (visible in the log), but does not work:
command: !Sub
- './install.sh -i ${GroupName}-access-group'
- Group: !Ref GroupName
This does not even parse ok:
command: !Sub './install.sh -i ${GroupName}-access-group'
Without Sub, it works.... e.g. command: './install.sh'
If I move the install.sh from the CF init command to the end in the UserData directly, it also works:
UserData:
....
${UbuntuInit}
/opt/install.sh -v -i ${GroupName}-access-group
Currently I have chosen for the following workaround:
UserData:
'Fn::Base64': !Sub
- |
#!/bin/bash
set -x
exec > >(tee /var/log/user-data.log|logger -t user-data ) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'
export REGION=${AWS::Region}
export STACKNAME=${AWS::StackName}
export GROUP=${GROUP}
${UserData}
- UserData: !FindInMap [UserDataMap, Ubuntu, UserData]
GROUP: !Ref GroupName
And in the UserData Map:
Ubuntu:
UserData: |
apt-get update
apt-get -y install python-setuptools
mkdir aws-cfn-bootstrap-latest
curl -s -m 60 https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1
easy_install aws-cfn-bootstrap-latest
/usr/local/bin/cfn-init --verbose --stack ${STACKNAME} --region ${REGION} --resource BastionLaunchConfig
/opt/install.sh -v -i ${GROUP}-access-group
/usr/local/bin/cfn-signal -e
So I pass an environment variable to the Map and call install.sh directly (not within CF init command)...
can you share the cloudformation template you use? at leats launchconfig, autoscalingroup and iam role
Sure,
- UserDataMap
See here that I call install.sh explicitly because Sub within a cfn-init command is not working for me.
Also in the Log, somehow the sshd restart is not done. But I also dont see any errors...
So I also do an ssh restart.
UserDataMap:
Ubuntu:
UserData: |
apt-get update
apt-get -y install python-setuptools
mkdir aws-cfn-bootstrap-latest
curl -s -m 60 https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1
easy_install aws-cfn-bootstrap-latest
/usr/local/bin/cfn-init --verbose --stack ${STACKNAME} --region ${REGION} --resource BastionLaunchConfig
/opt/install.sh -v -r v1.9.0 -i i2d-${GROUP}-redshift-access-group
service ssh restart
/usr/local/bin/cfn-signal -e 0 --stack ${STACKNAME} --region ${REGION} --resource BastionAutoScalingGroup
- IAM
BastionIamRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
BastionIamRoleIamPolicies:
Type: AWS::IAM::Policy
Properties:
PolicyName:
Fn::Join:
- ''
- - Ref: AWS::StackName
- "-instance-policy"
PolicyDocument:
Statement:
- Effect: Allow
Action:
- 'iam:ListUsers'
- 'iam:GetGroup'
Resource: '*'
- Effect: Allow
Action:
- 'iam:ListSSHPublicKeys'
- 'iam:GetSSHPublicKey'
Resource: !Sub 'arn:aws:iam::${AWS::AccountId}:user/*'
- Effect: Allow
Action:
- s3:*
- ec2:Describe*
- ssm:UpdateInstanceInformation
- elasticloadbalancing:Describe*
- autoscaling:Describe*
- cloudwatch:*
- logs:*
- sns:*
Resource: "*"
Roles:
- Ref: BastionIamRole
BastionIamInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: "/"
Roles:
- Ref: BastionIamRole
- LaunchConfiguration
See the GROUP export.
BastionLaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
DependsOn:
- IngressSecurityGroup
- BastionIamInstanceProfile
Metadata:
'AWS::CloudFormation::Init':
configSets:
default: [prepareUbuntu, install]
prepareUbuntu:
packages:
apt:
git: []
awscli: []
install:
files:
'/opt/install.sh':
source: 'https://raw.githubusercontent.com/widdix/aws-ec2-ssh/master/install.sh'
mode: '000755'
owner: root
group: root
# commands:
# a4_test:
# command: !Sub
# - 'echo ${IAMGROUP} >> group4.txt'
# - IAMGROUP: !Ref BastionUserGroup
# cwd: '/opt'
# a5_test:
# command: !Sub
# - 'echo i2d-${Group}-redshift-access-group >> group5.txt'
# - Group: !Ref GroupName
# cwd: '/opt'
# b_install:
# command: !Sub
# - './install.sh -v -i i2d-${Group}-redshift-access-group'
# - Group: !Ref GroupName
# cwd: '/opt'
Properties:
KeyName: !If [HasKeyName, !Ref KeyName, !Ref 'AWS::NoValue']
ImageId:
Ref: BastionAmi
InstanceMonitoring: 'true'
IamInstanceProfile:
Ref: BastionIamInstanceProfile
InstanceType:
Ref: BastionInstanceType
SecurityGroups:
- Ref: IngressSecurityGroup
UserData:
'Fn::Base64': !Sub
- |
#!/bin/bash
set -x
exec > >(tee /var/log/user-data.log|logger -t user-data ) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'
export REGION=${AWS::Region}
export STACKNAME=${AWS::StackName}
export GROUP=${GROUP}
${UserData}
- UserData: !FindInMap [UserDataMap, Ubuntu, UserData]
GROUP: !Ref GroupName
- AutoScalingGroup
BastionAutoScalingGroup:
Type: 'AWS::AutoScaling::AutoScalingGroup'
DependsOn: BastionLaunchConfig
Properties:
DesiredCapacity: 1
LaunchConfigurationName: !Ref BastionLaunchConfig
LoadBalancerNames:
- !Sub '${GroupName}-BastionLB'
MaxSize: 1
MinSize: 1
DesiredCapacity: 1
Tags:
- Key: Name
Value: !Sub '${GroupName}-Bastion'
PropagateAtLaunch: true
VPCZoneIdentifier:
- Ref: PublicSubnet1
- Ref: PublicSubnet2
Cooldown: '600'
CreationPolicy:
ResourceSignal:
Count: 1
Timeout: PT10M
UpdatePolicy:
AutoScalingRollingUpdate:
MaxBatchSize: 1
PauseTime: PT10M
SuspendProcesses:
- HealthCheck
- ReplaceUnhealthy
- AZRebalance
- AlarmNotification
- ScheduledActions
WaitOnResourceSignals: true
there hasn't been any progress for a while. feel free to reopen if that changes.
what was the result of your investigation?