aws-solutions/efs-backup

The EC2 instance was unable to find the mount IP OR mount EFS

Closed this issue · 2 comments

fadywageehsaad commented

I made a cloudformation template for the prerequisites, and here is the template:

AWSTemplateFormatVersion: '2010-09-09'
Description: Build Apache server, AutoScaling group, ELB and handle two different domain names
Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.40.0.0/16
      Tags:
      - Key: Application
        Value:
          Ref: AWS::StackId
      - Key: Name
        Value: FWagehPublicVPC
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
      - Key: Application
        Value:
          Ref: AWS::StackId
      - Key: Network
        Value: Public
  GatewayToInternet:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId:
        Ref: VPC
      InternetGatewayId:
        Ref: InternetGateway
  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: VPC
      CidrBlock: 10.40.0.0/24
      AvailabilityZone: !Select 
        - 1
        - Fn::GetAZs: !Ref 'AWS::Region'
      Tags:
      - Key: Application
        Value:
          Ref: AWS::StackId
      - Key: Name
        Value: fwageh_PublicSubnet
    PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
      Tags:
      - Key: Application
        Value:
          Ref: AWS::StackId
      - Key: Name
        Value: PublicRouteTable
  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId:
        Ref: PublicSubnet
      RouteTableId:
        Ref: PublicRouteTable
  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: GatewayToInternet
    Properties:
      RouteTableId:
        Ref: PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: InternetGateway
  PrivateSubnetMountOne:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: VPC
      CidrBlock: 10.40.3.0/24
      AvailabilityZone: !Select 
        - 0
        - Fn::GetAZs: !Ref 'AWS::Region'
      Tags:
      - Key: Application
        Value:
          Ref: AWS::StackId
      - Key: Name
        Value: fwageh_PrivateSubnetMountOne
  PrivateSubnetMountTwo:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: VPC
      CidrBlock: 10.40.4.0/24
      AvailabilityZone: !Select 
        - 1
        - Fn::GetAZs: !Ref 'AWS::Region'
      Tags:
      - Key: Application
        Value:
          Ref: AWS::StackId
      - Key: Name
        Value: fwageh_PrivateSubnetMountTwo
  NATGateway:
    DependsOn: GatewayToInternet
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId:
        Fn::GetAtt:
        - ElasticIP
        - AllocationId
      SubnetId:
        Ref: PublicSubnet
  ElasticIP:
    Type: AWS::EC2::EIP
    DependsOn: GatewayToInternet
    Properties:
      Domain: vpc
  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC
  PrivateRouteToInternet:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId:
        Ref: PrivateRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId:
        Ref: NATGateway
  PrivateSubnetOneRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId:
        Ref: PrivateSubnetMountOne
      RouteTableId:
        Ref: PrivateRouteTable
  PrivateSubnetTwoRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId:
        Ref: PrivateSubnetMountTwo
      RouteTableId:
        Ref: PrivateRouteTable
  FileSystem:
    Type: 'AWS::EFS::FileSystem'
    Properties:
      Encrypted: true
      KmsKeyId: !GetAtt 
        - key
        - Arn
  key:
    Type: 'AWS::KMS::Key'
    Properties:
      KeyPolicy:
        Version: 2012-10-17
        Id: key-default-1
        Statement:
          - Sid: Allow administration of the key
            Effect: Allow
            Principal:
              AWS: !Join 
                - ''
                - - 'arn:aws:iam::'
                  - !Ref 'AWS::AccountId'
                  - ':root'
            Action:
              - 'kms:*'
            Resource: '*'

And the SNS email Keeps telling me "The EFS backup was unsuccessful. The EC2 instance was unable to find the mount IP OR mount EFS"

And here is the ec2-log in my bucket:

Cloud-init v. 0.7.6 running 'init-local' at Mon, 30 Jul 2018 17:03:21 +0000. Up 6.78 seconds.
Cloud-init v. 0.7.6 running 'init' at Mon, 30 Jul 2018 17:03:21 +0000. Up 6.94 seconds.
ci-info: +++++++++++++++++++++++Net device info+++++++++++++++++++++++
ci-info: Device Up Address Mask Hw-Address
ci-info: lo True 127.0.0.1 255.0.0.0 .
ci-info: eth0 True 10.40.4.166 255.255.255.0 12:02:1a:0d:9b:d2
ci-info: ++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++
ci-info: Route Destination Gateway Genmask Interface Flags
ci-info: 0 0.0.0.0 10.40.4.1 0.0.0.0 eth0 UG
ci-info: 1 10.40.4.0 0.0.0.0 255.255.255.0 eth0 U
ci-info: 2 169.254.169.254 0.0.0.0 255.255.255.255 eth0 UH
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:vhv8fVQpM7DBmFLIXVDl/iaXTKB9uIzQK9xAWpvbJxQ root@ip-10-40-4-166
The key's randomart image is:
+---[RSA 2048]----+
| . +oBo.. |
| + + +. |
| .o E+o .|
| + +.=+o..|
| .S= + +++ |
| o. B + B .|
| ++ * * * |
| +..o = |
| o.. .. |
+----[SHA256]-----+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
SHA256:izZkrNY292nTQxNM/XUdOPRArX2TdpwxjquvWG/R9qs root@ip-10-40-4-166
The key's randomart image is:
+---[DSA 1024]----+
| o=o.o|
| .o+++|
| o +B|
| . oo B=|
| + S .+ +|
| = . . oo o |
| o B o +..o .|
| . o + .=o+. .|
| ooo+E...|
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
The key fingerprint is:
SHA256:9JT5t0+TVY3nfs/irvWGs28V4frzuWmVzmKxismTQ8k root@ip-10-40-4-166
The key's randomart image is:
+---[ECDSA 256]---+
| |
| o o.|
| . + o =|
| . o . =.|
| S o . o =|
| E + +=|
| . . O+|
| .+o =+@o|
| +o.++*@@|
+----[SHA256]-----+
Cloud-init v. 0.7.6 running 'modules:config' at Mon, 30 Jul 2018 17:03:22 +0000. Up 7.60 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper
1 package(s) needed (+0 related) for security, out of 3 available
Resolving Dependencies
--> Running transaction check
---> Package gnupg2.x86_64 0:2.0.28-2.31.amzn1 will be updated
---> Package gnupg2.x86_64 0:2.0.28-2.32.amzn1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size

Updating:
gnupg2 x86_64 2.0.28-2.32.amzn1 amzn-updates 2.6 M

Transaction Summary

Upgrade 1 Package

Total download size: 2.6 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : gnupg2-2.0.28-2.32.amzn1.x86_64 1/2
Cleanup : gnupg2-2.0.28-2.31.amzn1.x86_64 2/2
Verifying : gnupg2-2.0.28-2.32.amzn1.x86_64 1/2
Verifying : gnupg2-2.0.28-2.31.amzn1.x86_64 2/2

Updated:
gnupg2.x86_64 0:2.0.28-2.32.amzn1

Complete!
Cloud-init v. 0.7.6 running 'modules:final' at Mon, 30 Jul 2018 17:03:27 +0000. Up 13.28 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper
Existing lock /var/run/yum.pid: another copy is running as pid 2061.
Another app is currently holding the yum lock; waiting for it to exit...
The other application is: yum
Memory : 87 M RSS (334 MB VSZ)
Started: Mon Jul 30 17:03:26 2018 - 00:02 ago
State : Running, pid: 2061
Examining /var/tmp/yum-root-uoTNk3/amazon-ssm-agent.rpm: amazon-ssm-agent-2.2.800.0-1.x86_64
Marking /var/tmp/yum-root-uoTNk3/amazon-ssm-agent.rpm as an update to amazon-ssm-agent-2.2.120.0-1.amzn1.x86_64
Resolving Dependencies
--> Running transaction check
---> Package amazon-ssm-agent.x86_64 0:2.2.120.0-1.amzn1 will be updated
---> Package amazon-ssm-agent.x86_64 0:2.2.800.0-1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size

Updating:
amazon-ssm-agent x86_64 2.2.800.0-1 /amazon-ssm-agent 40 M

Transaction Summary

Upgrade 1 Package

Total size: 40 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
amazon-ssm-agent stop/waiting
Updating : amazon-ssm-agent-2.2.800.0-1.x86_64 1/2
Cleanup : amazon-ssm-agent-2.2.120.0-1.amzn1.x86_64 2/2
amazon-ssm-agent start/running, process 2183
Verifying : amazon-ssm-agent-2.2.800.0-1.x86_64 1/2
Verifying : amazon-ssm-agent-2.2.120.0-1.amzn1.x86_64 2/2

Updated:
amazon-ssm-agent.x86_64 0:2.2.800.0-1

Complete!
start: Job is already running: amazon-ssm-agent
--2018-07-30 17:03:36-- https://s3.amazonaws.com/solutions-reference/efs-backup/latest/efs-ec2-backup.sh
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.104.85
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.104.85|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3268 (3.2K) [binary/octet-stream]
Saving to: ‘/home/ec2-user/efs-ec2-backup.sh’

 0K ...                                                   100% 10.5M=0s

2018-07-30 17:03:36 (10.5 MB/s) - ‘/home/ec2-user/efs-ec2-backup.sh’ saved [3268/3268]

--2018-07-30 17:03:36-- https://s3.amazonaws.com/solutions-reference/efs-backup/latest/efs-backup-fpsync.sh
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.168.157
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.168.157|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5903 (5.8K) [binary/octet-stream]
Saving to: ‘/home/ec2-user/efs-backup-fpsync.sh’

 0K .....                                                 100% 11.2M=0.001s

2018-07-30 17:03:36 (11.2 MB/s) - ‘/home/ec2-user/efs-backup-fpsync.sh’ saved [5903/5903]

�[H�[JThis is the master script to perform efs backup

input from user

_source_efs: fs-d0464798
_destination_efs: fs-5b7f7e13
_interval: daily
_retain: 7
_folder_label: fwageh-efs-backup
_backup_prefix: /
region is us-east-1
instance-id is i-0582c39e3a29996e5
-- 2018-07-30T17:03:38 -- resolving source efs address fs-d0464798.efs.us-east-1.amazonaws.com
10.40.4.202
-- 2018-07-30T17:03:38 -- src mount ip: 10.40.4.202
-- 2018-07-30T17:03:38 -- resolving backup efs address fs-5b7f7e13.efs.us-east-1.amazonaws.com
10.40.4.200
-- 2018-07-30T17:03:38 -- dst mount ip: 10.40.4.200
-- 2018-07-30T17:03:38 -- running EFS backup script

input from user

source: 10.40.4.202:/
destination: 10.40.4.200:/
interval: daily
retain: 7
efsid: fwageh-efs-backup
-- 2018-07-30T17:03:38 -- sudo yum -y update
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package amazon-ssm-agent.x86_64 0:2.2.800.0-1 will be updated
---> Package amazon-ssm-agent.x86_64 0:2.2.800.0-1.amzn1 will be an update
---> Package kernel.x86_64 0:4.14.55-62.37.amzn1 will be installed
---> Package kernel-tools.x86_64 0:4.14.47-56.37.amzn1 will be updated
---> Package kernel-tools.x86_64 0:4.14.55-62.37.amzn1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size

Installing:
kernel x86_64 4.14.55-62.37.amzn1 amzn-updates 21 M
Updating:
amazon-ssm-agent x86_64 2.2.800.0-1.amzn1 amzn-updates 12 M
kernel-tools x86_64 4.14.55-62.37.amzn1 amzn-updates 124 k

Transaction Summary

Install 1 Package
Upgrade 2 Packages

Total download size: 33 M
Downloading packages:

Total 16 MB/s | 33 MB 00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kernel-4.14.55-62.37.amzn1.x86_64 1/5
Updating : kernel-tools-4.14.55-62.37.amzn1.x86_64 2/5
Updating : amazon-ssm-agent-2.2.800.0-1.amzn1.x86_64 3/5
Cleanup : kernel-tools-4.14.47-56.37.amzn1.x86_64 4/5
Cleanup : amazon-ssm-agent-2.2.800.0-1.x86_64 5/5
Verifying : amazon-ssm-agent-2.2.800.0-1.amzn1.x86_64 1/5
Verifying : kernel-tools-4.14.55-62.37.amzn1.x86_64 2/5
Verifying : kernel-4.14.55-62.37.amzn1.x86_64 3/5
Verifying : kernel-tools-4.14.47-56.37.amzn1.x86_64 4/5
Verifying : amazon-ssm-agent-2.2.800.0-1.x86_64 5/5

Installed:
kernel.x86_64 0:4.14.55-62.37.amzn1

Updated:
amazon-ssm-agent.x86_64 0:2.2.800.0-1.amzn1
kernel-tools.x86_64 0:4.14.55-62.37.amzn1

Complete!
-- 2018-07-30T17:04:08 -- sudo yum -y install nfs-utils
Loaded plugins: priorities, update-motd, upgrade-helper
Existing lock /var/run/yum.pid: another copy is running as pid 7603.
Another app is currently holding the yum lock; waiting for it to exit...
The other application is: yum
Memory : 35 M RSS (282 MB VSZ)
Started: Mon Jul 30 17:04:08 2018 - 00:00 ago
State : Running, pid: 7603
Package 1:nfs-utils-1.3.0-0.21.amzn1.x86_64 already installed and latest version
Nothing to do
-- 2018-07-30T17:04:10 -- sudo mkdir /backup
-- 2018-07-30T17:04:10 -- sudo mkdir /mnt/backups
-- 2018-07-30T17:04:10 -- sudo mount -t nfs -o nfsvers=4.1 -o rsize=1048576 -o wsize=1048576 -o timeo=600 -o retrans=2 -o hard 10.40.4.202:/ /backup
mount.nfs: Connection timed out
mount status for source efs: 32
-- 2018-07-30T17:08:33 -- sudo mount -t nfs -o nfsvers=4.1 -o rsize=1048576 -o wsize=1048576 -o timeo=600 -o retrans=2 -o hard 10.40.4.200:/ /mnt/backups
mount status for backup efs: 0
-- 2018-07-30T17:08:36 -- ERROR:efs_not_mounted
-- 2018-07-30T17:08:36 -- Backup script finished before the backup window, stopping the ec2 instance.
ci-info: no authorized ssh keys fingerprints found for user ec2-user.
Cloud-init v. 0.7.6 finished at Mon, 30 Jul 2018 17:08:37 +0000. Datasource DataSourceEc2. Up 322.79 seconds

I don't know where is the issue

gsingh04 commented

mount.nfs: Connection timed out
mount status for source efs: 32
-- 2018-07-30T17:08:33 -- sudo mount -t nfs -o nfsvers=4.1 -o rsize=1048576 -o wsize=1048576 -o timeo=600 -o retrans=2 -o hard 10.40.4.200:/ /mnt/backups
mount status for backup efs: 0
-- 2018-07-30T17:08:36 -- ERROR:efs_not_mounted

we can see the EFS was not mounted. please refer to this link to understand how to allow access to EFS mount targets. you must allow NFS port in your inbound rules so EC2 can mount the EFS volume.

fadywageehsaad commented

Thanks @gsingh04 ,
I found two issues:
1 - I should enable hostnames in the VPC so I added this line to the VPC:
EnableDnsSupport: true

VPC:
    Type: AWS::EC2::VPC
     Properties:
         CidrBlock: 10.40.0.0/16
         EnableDnsSupport: true
         Tags:
         - Key: Application
            Value:
                Ref: AWS::StackId
         - Key: Name
            Value: FWagehPublicVPC

2 - As @gsingh04 said I should allow NFS port in your inbound rules so EC2 can mount the EFS
volume, but I didn't find in the parameters any parametewre for input secuirty group for the EC2
instance. So added one like this:

         EC2SecuirtyGroup:
              Type: AWS::EC2::SecurityGroup
              Properties:
                  GroupDescription: Allow access 
                  VpcId:
                      Ref: VPC
                  SecurityGroupIngress:
                  - IpProtocol: tcp
                     FromPort: '22'
                     ToPort: '22'
                     CidrIp: 0.0.0.0/0

         MountTargetSecuirtyGroup:
              Type: AWS::EC2::SecurityGroup
               Properties:
                   GroupDescription: Allow access from load balancer
                   VpcId:
                     Ref: VPC
                   SecurityGroupIngress:
                   - IpProtocol: tcp
                      FromPort: '2049'
                      ToPort: '2049'
                      SourceSecurityGroupId:
                            Ref: EC2SecuirtyGroup