[service] [request]: Respect empty string variables set via `environmentFiles` in Fargate

[mxxk]

Note

This GitHub issue corresponds to AWS Support Case 172722539600462. AWS Support advised me to open this GitHub issue for public visibility and tracking.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

ECS Fargate support for .env files subtly differs from that of Docker. Specifically, empty environment variable entries (e.g., EMPTY=) are propagated via docker run --env-file, are ignored within Fargate ECS. ECS documentation states that environmentFiles behaves like docker run --env-file, so this request is to alleviate this discrepancy within Fargate ECS.

Which service(s) is this request for?

This is observed on Fargate ECS. (It possible also affects Fargate EKS, I did not test it.)

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

ECS documentation on environmentFiles seems to convey that the behavior of environmentFiles on Fargate matches that of Docker:

The following rules apply to the Fargate launch type:

The file is handled similar to a native Docker env-file.

However, the way ECS Fargate's handling of environmentFiles differs from Docker's handling of --env-file (see reproduction steps below).

Are you currently working around this issue?

This problem affects environmentFiles, but not environment. It is possible to pass an empty environment variable via environment:

"environment": [
    {
        "name": "VAR_EMPTY",
        "value": ""
    }
]

Additional context

Step-by-step reproduction

Docker behavior

1. Create the following file (vars.env) which defines two environment variables:

   VAR_NONEMPTY=value
   VAR_EMPTY=
   

2. Verify that docker run propagates both variables to the container:

   $ docker run --rm --env-file vars.env alpine sh -c 'env | grep ^VAR_'
   VAR_NONEMPTY=value
   VAR_EMPTY=
   

ECS behavior

1. Upload vars.env to S3 (s3://ecs-env-file-test/vars.env).

2. Create a new IAM role to use as an ECS task execution role (EcsEnvFileTestExecutionRole).

3. Allow ECS to assume EcsEnvFileTestExecutionRole via the following trust policy:

   {
       "Version": "2012-10-17",
       "Statement": {
           "Effect": "Allow",
           "Principal": {
               "Service": "ecs-tasks.amazonaws.com"
           },
           "Action": "sts:AssumeRole"
       }
   }

4. Allow EcsEnvFileTestExecutionRole to perform basic ECS execution role tasks by adding the AWS-managed policy AmazonECSTaskExecutionRolePolicy to its permissions.

5. Allow EcsEnvFileTestExecutionRole to read s3://ecs-env-file-test/vars.env by adding the following inline policy to its permissions:

   {
       "Version": "2012-10-17",
       "Statement": {
           "Effect": "Allow",
           "Action": "s3:GetObject",
           "Resource": "arn:aws:s3:::ecs-env-file-test/vars.env"
       }
   }

6. Create the default ECS cluster.

7. Create a CloudWatch Logs group (/ecs/EnvFileTest) for ECS to write logs to.

8. Create ECS task definition (EnvFileTest) with JSON:

   {
       "requiresCompatibilities": [
           "FARGATE"
       ],
       "family": "EnvFileTest",
       "containerDefinitions": [
           {
               "name": "alpine",
               "image": "alpine",
               "command": [
                   "sh",
                   "-c",
                   "env | grep ^VAR_"
               ],
               "environmentFiles": [
                   {
                       "value": "arn:aws:s3:::ecs-env-file-test/vars.env",
                       "type": "s3"
                   }
               ],
               "logConfiguration": {
                   "logDriver": "awslogs",
                   "options": {
                       "awslogs-create-group": "true",
                       "awslogs-group": "/ecs/EnvFileTest",
                       "awslogs-region": "us-west-2",
                       "awslogs-stream-prefix": "ecs"
                   }
               }
           }
       ],
       "networkMode": "awsvpc",
       "memory": "512",
       "cpu": "256",
       "executionRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/EcsEnvFileTestExecutionRole"
   }

9. Run task definition EnvFileTest on Fargate on the default ECS cluster. For task networking options, choose the default VPC and assign the default security group. Keep the "Public IP" toggle on to allow ECS to pull images from the Docker Hub registry and fetch the environment file from S3 (which is simpler than the alternative option of setting up VPC endpoints).

10. Examine the CloudWatch logs from the container, and notice that the environment variable VAR_EMPTY is missing:

    VAR_NONEMPTY=value
    

    This differs from the behavior of docker run --env-file.

11. (Side point) If the same ECS task is run on EC2 instead of Fargate, VAR_EMPTY is present (which matches docker run --env-file). I did not try ECS Anywhere (EXTERNAL deployment) option. AWS Support kindly pointed out that this is because of acommunity contribution: aws/amazon-ecs-agent#3797