Lambda layer build with pip requirements.txt works for local execution but fails using docker

Question

Lambda layer build with pip requirements.txt works for local execution but fails using docker

caioquirino opened this issue a year ago · 14 comments

Description

Please provide a clear and concise description of the issue you are encountering, and a reproduction of your configuration (see the examples/* directory for references that you can copy+paste and tailor to match your configs if you are unable to copy your exact configuration). The reproduction MUST be executable by running terraform init && terraform apply without any further changes.

If your request is for a new feature, please use the Feature request template.

✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following first:

Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
Re-initialize the project root to pull down modules: terraform init
Re-attempt your terraform plan or apply and check if the issue still persists

Versions

Module version [Required]: 4.17.0
Terraform version: 1.4.2

Provider version(s): 4.65.0

Reproduction Code [Required]

Code:

locals {
  build_in_docker = true
}
module "lambda_layer" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 4.17.0"

  create_layer        = true
  layer_name          = "layer"
  compatible_runtimes = ["python3.9"]
  architectures       = ["x86_64"]

  store_on_s3   = true
  s3_bucket     = "bucket"
  s3_prefix     = "layers/"

  source_path = [
    "${path.root}/../requirements.txt",
    {
      pip_requirements = "${path.root}/../requirements.txt"
    },
  ]

  build_in_docker           = local.build_in_docker
  docker_pip_cache          = local.build_in_docker
  docker_additional_options = local.build_in_docker ? ["--platform", "linux/amd64"] : null
  runtime                   = "python3.9"
}

Files:

requirements.txt
localstack/
            main.tf
            ....tf

Execution: cd localstack && terraform apply -auto-approve

Steps to reproduce the behavior:

When I run the code above following the folder structure above, it works normally when building locally, but it builds a package with macos binaries. In order to build it for linux, I have to use the docker setup. But when I try it, it doesn't work. It throws an error instead, saying that there's no requirements.txt file.

Not using workspaces, I have deleted the terraform state and .terraform dirs entirely, as well as recreated the localstack instance.

Expected behavior

The lambda layer's artifact should be built properly from the requirements.txt file

Actual behavior

It failed complaining that the requirements.txt file does not exist

Terminal Output Screenshot(s)

Additional context

│ Error: local-exec provisioner error
│ 
│   with module.lambda_layer_poetry.null_resource.archive[0],
│   on .terraform/modules/lambda_layer_poetry/package.tf line 67, in resource "null_resource" "archive":
│   67:   provisioner "local-exec" {
│ 
│ Error running command 'builds/be89b46f2726a9d44b2733a4b9c40a5d1717ab38c2c299851d2ad7dee97ac085.plan.json': exit status 1. Output: zip: creating 'builds/be89b46f2726a9d44b2733a4b9c40a5d1717ab38c2c299851d2ad7dee97ac085.zip'
│ archive
│ zip: adding: requirements.txt
│ Installing python requirements: ./../requirements.txt
│ > mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/sb/hzwg60vj2l391dqngh0l0swm0000gp/T/terraform-aws-lambda-5pyvm69o
│ > cd /var/folders/sb/hzwg60vj2l391dqngh0l0swm0000gp/T/terraform-aws-lambda-5pyvm69o
│ > docker run --rm -w /var/task -v /private/var/folders/sb/hzwg60vj2l391dqngh0l0swm0000gp/T/terraform-aws-lambda-5pyvm69o:/var/task:z -v /Users/caio.quirinodasilva@TMNL.nl/.ssh/known_hosts:/root/.ssh/known_hosts:z --platform
│ linux/amd64 -v /Users/caio.quirinodasilva@TMNL.nl/github/tmnlorg/digital-channel-data-ingestion/localstack/builds/cache/pip:/root/.cache/pip:z --entrypoint '' public.ecr.aws/sam/build-python3.9 /bin/sh -c 'python3.9 -m pip
│ install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 502:20 .'
│ ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
│ WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
│ You should consider upgrading via the '/var/lang/bin/python3.9 -m pip install --upgrade pip' command.
│ zip: Error during zip archive creation
│ Traceback (most recent call last):
│   File "/Users/caio.quirinodasilva@TMNL.nl/github/tmnlorg/digital-channel-data-ingestion/localstack/.terraform/modules/lambda_layer_poetry/package.py", line 1516, in build_command
│     bpm.execute(build_plan, zs, query)
│   File "/Users/caio.quirinodasilva@TMNL.nl/github/tmnlorg/digital-channel-data-ingestion/localstack/.terraform/modules/lambda_layer_poetry/package.py", line 859, in execute
│     with install_pip_requirements(query, pip_requirements, tmp_dir) as rd:
│   File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/contextlib.py", line 117, in __enter__
│     return next(self.gen)
│   File "/Users/caio.quirinodasilva@TMNL.nl/github/tmnlorg/digital-channel-data-ingestion/localstack/.terraform/modules/lambda_layer_poetry/package.py", line 1004, in install_pip_requirements
│     check_call(docker_run_command(
│   File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
│     raise CalledProcessError(retcode, cmd)
│ subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-w', '/var/task', '-v', '/private/var/folders/sb/hzwg60vj2l391dqngh0l0swm0000gp/T/terraform-aws-lambda-5pyvm69o:/var/task:z', '-v',
│ '/Users/caio.quirinodasilva@TMNL.nl/.ssh/known_hosts:/root/.ssh/known_hosts:z', '--platform', 'linux/amd64', '-v',
│ '/Users/caio.quirinodasilva@TMNL.nl/github/tmnlorg/digital-channel-data-ingestion/localstack/builds/cache/pip:/root/.cache/pip:z', '--entrypoint', '', 'public.ecr.aws/sam/build-python3.9', '/bin/sh', '-c', 'python3.9 -m pip
│ install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 502:20 .']' returned non-zero exit status 1.

punkrokk commented a year ago

same here

Answer 1 · 2023-06-08T20:50:01.000Z

Hi, today I encountered the exact same error. I use a MacBook Pro with an Apple M1 Pro chip and macOS Ventura 13.4. I'm using Rancher Desktop, and even though I prefer containerd+nerdctl as the container engine, I have configured dockerd (moby) for the comment below. Also, I haven't delved into this module's specifics, so my analysis may be incorrect and potentially misleading.

Terraform:

module "audit_exporter" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 5.0.0"
 
  function_name = "${local.project}-eks-audit-exporter"
  description   = "Export EKS audit logs from CloudWatch and uploads them to S3"
  handler       = "app.lambda_handler"
  runtime       = "python3.8"
  architectures = ["x86_64"]
 
  source_path = [
    {
      path             = "${path.root}/assets/audit_exporter_lambda"
      pip_requirements = true
      patterns         = ["!.venv/.*"]
    }
  ]
 
  build_in_docker   = true
  docker_additional_options = ["--platform", "linux/amd64"]
 
  store_on_s3 = true
  s3_bucket   = module.audit_exporter_artifacts.s3_bucket_id
 
  environment_variables = {
    BUCKET_NAME = "mybucket"
  }
}

Output:

module.audit_exporter.null_resource.archive[0]: Provisioning with 'local-exec'...
module.audit_exporter.null_resource.archive[0] (local-exec): Executing: ["python3" ".terraform/modules/audit_exporter/package.py" "build" "--timestamp" "1686251027549448000" "builds/bcd093d772e6d337d36210bded0729f72b6a3e1a74bd3167093b73f0de4ad58b.plan.json"]
module.audit_exporter.null_resource.archive[0] (local-exec): zip: creating 'builds/bcd093d772e6d337d36210bded0729f72b6a3e1a74bd3167093b73f0de4ad58b.zip' archive
module.audit_exporter.null_resource.archive[0] (local-exec): Installing python requirements: ./assets/audit_exporter_lambda/requirements.txt
module.audit_exporter.null_resource.archive[0] (local-exec): > mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_
module.audit_exporter.null_resource.archive[0] (local-exec): > cd /var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_
module.audit_exporter.null_resource.archive[0] (local-exec): > docker run --rm -w /var/task -v /private/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_:/var/task:z -v /Users/xxxxxxxxxxxxxx/.ssh/known_hosts:/root/.ssh/known_hosts:z --platform linux/amd64 --entrypoint '' public.ecr.aws/sam/build-python3.8 /bin/sh -c 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 501:20 .'
module.audit_exporter.null_resource.archive[0] (local-exec): ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
module.audit_exporter.null_resource.archive[0] (local-exec): WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
module.audit_exporter.null_resource.archive[0] (local-exec): You should consider upgrading via the '/var/lang/bin/python3.8 -m pip install --upgrade pip' command.
module.audit_exporter.null_resource.archive[0] (local-exec): zip: Error during zip archive creation
module.audit_exporter.null_resource.archive[0] (local-exec): Traceback (most recent call last):
module.audit_exporter.null_resource.archive[0] (local-exec):   File ".terraform/modules/audit_exporter/package.py", line 1516, in build_command
module.audit_exporter.null_resource.archive[0] (local-exec):     bpm.execute(build_plan, zs, query)
module.audit_exporter.null_resource.archive[0] (local-exec):   File ".terraform/modules/audit_exporter/package.py", line 859, in execute
module.audit_exporter.null_resource.archive[0] (local-exec):     with install_pip_requirements(query, pip_requirements, tmp_dir) as rd:
module.audit_exporter.null_resource.archive[0] (local-exec):   File "/Users/xxxxxxxxxxxxxx/.asdf/installs/python/3.8.16/lib/python3.8/contextlib.py", line 113, in __enter__
module.audit_exporter.null_resource.archive[0] (local-exec):     return next(self.gen)
module.audit_exporter.null_resource.archive[0] (local-exec):   File ".terraform/modules/audit_exporter/package.py", line 1004, in install_pip_requirements
module.audit_exporter.null_resource.archive[0] (local-exec):     check_call(docker_run_command(
module.audit_exporter.null_resource.archive[0] (local-exec):   File "/Users/xxxxxxxxxxxxxx/.asdf/installs/python/3.8.16/lib/python3.8/subprocess.py", line 364, in check_call
module.audit_exporter.null_resource.archive[0] (local-exec):     raise CalledProcessError(retcode, cmd)
module.audit_exporter.null_resource.archive[0] (local-exec): subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-w', '/var/task', '-v', '/private/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_:/var/task:z', '-v', '/Users/xxxxxxxxxxxxxx/.ssh/known_hosts:/root/.ssh/known_hosts:z', '--platform', 'linux/amd64', '--entrypoint', '', 'public.ecr.aws/sam/build-python3.8', '/bin/sh', '-c', 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 501:20 .']' returned non-zero exit status 1.

Error:

Error: local-exec provisioner error
 
  with module.audit_exporter.null_resource.archive[0],
  on .terraform/modules/audit_exporter/package.tf line 67, in resource "null_resource" "archive":
  67:   provisioner "local-exec" {
 
Error running command 'builds/bcd093d772e6d337d36210bded0729f72b6a3e1a74bd3167093b73f0de4ad58b.plan.json': exit status 1. Output: zip: creating
'builds/bcd093d772e6d337d36210bded0729f72b6a3e1a74bd3167093b73f0de4ad58b.zip' archive
Installing python requirements: ./assets/audit_exporter_lambda/requirements.txt
> mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_
> cd /var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_
> docker run --rm -w /var/task -v /private/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_:/var/task:z -v
/Users/xxxxxxxxxxxxxx/.ssh/known_hosts:/root/.ssh/known_hosts:z --platform linux/amd64 --entrypoint '' public.ecr.aws/sam/build-python3.8 /bin/sh
-c 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 501:20 .'
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/var/lang/bin/python3.8 -m pip install --upgrade pip' command.
zip: Error during zip archive creation
Traceback (most recent call last):
  File ".terraform/modules/audit_exporter/package.py", line 1516, in build_command
    bpm.execute(build_plan, zs, query)
  File ".terraform/modules/audit_exporter/package.py", line 859, in execute
    with install_pip_requirements(query, pip_requirements, tmp_dir) as rd:
  File "/Users/xxxxxxxxxxxxxx/.asdf/installs/python/3.8.16/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File ".terraform/modules/audit_exporter/package.py", line 1004, in install_pip_requirements
    check_call(docker_run_command(
  File "/Users/xxxxxxxxxxxxxx/.asdf/installs/python/3.8.16/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-w', '/var/task', '-v',
'/private/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_:/var/task:z', '-v',
'/Users/xxxxxxxxxxxxxx/.ssh/known_hosts:/root/.ssh/known_hosts:z', '--platform', 'linux/amd64', '--entrypoint', '',
'public.ecr.aws/sam/build-python3.8', '/bin/sh', '-c', 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt
&& chown -R 501:20 .']' returned non-zero exit status 1.

By locally modifying package.py (which is neither a solution nor a workaround but useful for debugging purposes), I ensured that the requirements.txt file was correctly copied to the temporary folder:

    log.info('Installing python requirements: %s', requirements_file)
    with tempdir(tmp_dir) as temp_dir:
        requirements_filename = os.path.basename(requirements_file)
        target_file = os.path.join(temp_dir, requirements_filename)
        shutil.copyfile(requirements_file, target_file)
+       log.info(os.listdir(temp_dir))
 
        python_exec = runtime
        subproc_env = None
 
        if not docker:
            if WINDOWS:
                python_exec = 'python.exe'
            elif OSX:
                # Workaround for OSX when XCode command line tools'
                # python becomes the main system python interpreter
                os_path = '{}:/Library/Developer/CommandLineTools' \
                          '/usr/bin'.format(os.environ['PATH'])
                subproc_env = os.environ.copy()
                subproc_env['PATH'] = os_path
 
        # Install dependencies into the temporary directory.
        with cd(temp_dir):
+           log.info(os.listdir('.'))
            pip_command = [
                python_exec, '-m', 'pip',
                'install', '--no-compile',
                '--prefix=', '--target=.',
                '--requirement={}'.format(requirements_filename),
            ]

For both functions, I received module.audit_exporter.null_resource.archive[0] (local-exec): ['requirements.txt'] as the output.

I noticed that the bind mount is being performed on a formally different folder than the one created:

module.audit_exporter.null_resource.archive[0] (local-exec): > mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_
module.audit_exporter.null_resource.archive[0] (local-exec): > cd /var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_
module.audit_exporter.null_resource.archive[0] (local-exec): > docker run --rm -w /var/task -v /private/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/terraform-aws-lambda-qfm73ip_:/var/task:z -v /Users/matteocellucci/.ssh/known_hosts:/root/.ssh/known_hosts:z --platform linux/amd64 --entrypoint '' public.ecr.aws/sam/build-python3.8 /bin/sh -c 'python3.8 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 501:20 .'

Upon examining the folder, I noticed that it is actually a symlink:

$ ls -l /
total 10
[…]
drwxr-xr-x   6 root  wheel   192 May 31 09:27 private
[…]
lrwxr-xr-x@  1 root  wheel    11 May 13 00:29 var -> private/var

Although a symlink shouldn't cause any issues while researching online, I came across this article (I couldn't find more official sources): https://medium.com/effy-tech/fixing-the-var-folders-error-in-docker-for-mac-v2-2-3-2a40e776132d.

By still modifying the local package.py file, I managed to make the module work by changing the mounted path in the container:

        with cd(temp_dir):
            pip_command = [
                python_exec, '-m', 'pip',
                'install', '--no-compile',
                '--prefix=', '--target=.',
                '--requirement={}'.format(requirements_filename),
            ]
            if docker:
                with_ssh_agent = docker.with_ssh_agent
                pip_cache_dir = docker.docker_pip_cache
                if pip_cache_dir:
                    if isinstance(pip_cache_dir, str):
                        pip_cache_dir = os.path.abspath(
                            os.path.join(working_dir, pip_cache_dir))
                    else:
                        pip_cache_dir = os.path.abspath(os.path.join(
                            working_dir, artifacts_dir, 'cache/pip'))
 
                chown_mask = '{}:{}'.format(os.getuid(), os.getgid())
                shell_command = [shlex_join(pip_command), '&&',
                                 shlex_join(['chown', '-R',
                                             chown_mask, '.'])]
                shell_command = [' '.join(shell_command)]
                check_call(docker_run_command(
-                   '.', shell_command, runtime,
+                   temp_dir, shell_command, runtime,
                    image=docker_image_tag_id,
                    shell=True, ssh_agent=with_ssh_agent,
                    pip_cache_dir=pip_cache_dir, docker=docker,
                ))

Additionally, while entering the Python REPL, I conducted the following tests:

$ python
Python 3.8.16 (default, Jun  1 2023, 14:39:18)
[Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.chdir("/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T/")
>>> os.getcwd()
'/private/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T'
>>> os.path.abspath('.')
'/private/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T'
>>> os.system("pwd")
/private/var/folders/nr/n2kh28194_31djxs4n734hwr0000gn/T

I also tried enabling Administrative Access to Rancher Desktop without success.

EDIT#1: This problem may be related to number #137, albeit in different environments.

EDIT#2: Maybe I found a viable workaround. Setting pip_tmp_dir to a folder that does not suffer from the described issue like pip_tmp_dir = "${path.module}/.build". Unfortunately, I encountered another error. I'm investigating it and will let you know.

Answer 2 · 2023-06-09T09:57:57.000Z

Based on the analysis of the previous comment, I made it work by allowing the mount of /private/var/folders/. Following this comment on GitHub (rancher-sandbox/rancher-desktop#2875 (comment)), I created the file ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml and added the following content:

mounts:
  - location: /private/var/folders
    writable: true

After restarting Rancher Desktop, it is working correctly now. I don't know if this is a permanent solution or a workaround. Although I think this module should handle the default configuration of Rancher Desktop, I'm unsure of the best way to do it.

Answer 3 · 2023-07-10T00:21:05.000Z

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

Answer 4 · 2023-08-12T00:14:17.000Z

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

Answer 5 · 2023-09-11T13:40:36.000Z

Same issue here, does not work on Windows.
Any suggestion I could find for the poor guy on Windows was to use Unix based system 🤷‍♂️

Answer 6 · 2023-10-12T00:14:13.000Z

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

Answer 7 · 2023-10-12T06:32:55.000Z

Unstale...

Answer 8 · 2023-11-03T14:18:54.000Z

Please consider running examples/build-package - there are multiple similar samples which can help you to find out the right set of arguments.

PS: I can't reproduce the error with the provided code snippet.

Answer 9 · 2023-11-03T14:24:01.000Z

@antonbabenko Is there any chance that you try using Apple Silicon?

My scenario was exactly:

Compiling on Apple Silicon (macos-darwin) to run on Localstack (linux-x86-64)

Answer 10 · 2023-11-03T15:07:47.000Z

@caioquirino I was able to run the code provided by you (with valid path to requirements.txt) using tflocal apply (and verify with awslocal lambda list-layers).

module "package_file_with_pip_requirements" {
  source = "../../"   # <- using latest version

  create_function = false

  runtime = "python3.8"

  source_path = [
    "${path.module}/../fixtures/python3.8-app1/requirements.txt",
    {
      pip_requirements = "${path.module}/../fixtures/python3.8-app1/requirements.txt"
    },
  ]

  create_layer = true
  layer_name   = "layer2435"
  architectures    = ["x86_64"]
  build_in_docker  = true
  docker_pip_cache = true
}

It just works for me. :)

Answer 11 · 2023-12-04T00:17:25.000Z

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

Answer 12 · 2023-12-15T00:17:31.000Z

This issue was automatically closed because of stale in 10 days

Answer 13 · 2024-01-14T02:09:13.000Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.