Python - Allow "platform" option for pip installations
lukas-hetzenecker opened this issue · 4 comments
Is your request related to a new offering from AWS?
No
Is your request related to a problem? Please describe.
This request is based on this minimal example based on the terraform code
module "lambda_function" {
source = "terraform-aws-modules/lambda/aws"
version = "~> 3.1"
function_name = "lambda_function"
description = "Lambda function that does lambda stuff"
handler = "main.lambda_handler"
runtime = "python3.9"
source_path = "${path.module}/scripts/lambda_function"
}the handler function, and a requirements.txt file, containing its dependencies:
cryptography
So far, this is a quite standard lambda deployment of a python function.
When invoking this lambda function, it fails with the following error message:
[ERROR] Runtime.ImportModuleError: Unable to import module 'main': /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /var/task/cryptography/hazmat/bindings/_rust.abi3.so)
Traceback (most recent call last):
This hit us by surprise and took a while to pin down.
Our CI image running the terraform apply and assembling the lambda deployment package is based on a fairly new linux distribution. Therefore when pip runs, it selects a binary wheel archive for platform cp36-abi3-manylinux_2_28_x86_64, as this one is perfectly compatible with our build host.
But that does not mean this package will be compatible with the AWS Lambda runtime - the python3.9 runtime is based on a custom Amazon Linux 2 image. This image does not contain such a recent glibc release, resulting in the error shown above. To be binary compatible, another platform for the wheel would have to be selected.
A different example to showcase the issue is by running the above terraform code on my local Macbook.
Doing that results in the following action:
> terraform apply
[...]
# module.lambda_function.null_resource.archive[0] must be replaced
-/+ resource "null_resource" "archive" {
~ id = "4887585264796291062" -> (known after apply)
~ triggers = { # forces replacement
~ "timestamp" = "1663071319698200000" -> "1663072297750462000"
# (1 unchanged element hidden)
}
}
Plan: 1 to add, 0 to change, 1 to destroy.
[...]
module.lambda_function.null_resource.archive[0]: Creating...
module.lambda_function.null_resource.archive[0]: Provisioning with 'local-exec'...
module.lambda_function.null_resource.archive[0] (local-exec): Executing: ["python3" ".terraform/modules/lambda_function/package.py" "build" "--timestamp" "1663072297750462000" "builds/2576c817ebffdc69f333b98308abe4ce0389bb32513e67a9496393c2f0a3c5ae.plan.json"]
module.lambda_function.null_resource.archive[0] (local-exec): zip: creating 'builds/2576c817ebffdc69f333b98308abe4ce0389bb32513e67a9496393c2f0a3c5ae.zip' archive
module.lambda_function.null_resource.archive[0] (local-exec): Installing python requirements: ./scripts/lambda_function/requirements.txt
module.lambda_function.null_resource.archive[0] (local-exec): > mktemp -d terraform-aws-lambda-XXXXXXXX # /var/folders/kq/4kn3b591269d6xg0vc9x9qch0000gp/T/terraform-aws-lambda-5jumyzt5
module.lambda_function.null_resource.archive[0] (local-exec): > cd /var/folders/kq/4kn3b591269d6xg0vc9x9qch0000gp/T/terraform-aws-lambda-5jumyzt5
module.lambda_function.null_resource.archive[0] (local-exec): > python3.9 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt
module.lambda_function.null_resource.archive[0] (local-exec): Collecting cryptography
module.lambda_function.null_resource.archive[0] (local-exec): Using cached cryptography-38.0.1-cp36-abi3-macosx_10_10_x86_64.whl (2.8 MB)
[...]
Of importance here is this line: Using [...] cryptography-38.0.1-cp36-abi3-macosx_10_10_x86_64.whl
When not specified otherwise, pip defaults to collecting wheels compatible with the platform of the running system.
The running platform is macos - which will never be compatible with the lambda runtime.
Describe the solution you'd like.
Instead of defaulting to the platform of the running system, terraform-aws-lambda should default to a platform compatible with the lambda runtime when collecting binary packages.
According to the AWS documentation, this requires the --platform manylinux2014_x86_64 parameter for pip.
pip is called in package.py#L927, but there is no possibility to specify a platform parameter.
To solve this issue, either
a) terraform-aws-modules/lambda/aws should accept a platform parameter, which gets passed down to pip, or
b) contain sane defaults for python runtimes (manylinux2014_x86_64 ?)
Describe alternatives you've considered.
As a workaround, the definition could be changed to the following:
module "lambda_function" {
source = "terraform-aws-modules/lambda/aws"
version = "~> 3.1"
function_name = "lambda_function"
description = "Lambda function that does lambda stuff"
handler = "main.lambda_handler"
runtime = "python3.9"
source_path = [
{
path = "${path.module}/scripts/lambda_function"
commands = [
":zip",
"cd `mktemp -d`",
"python3.9 -m pip install --no-compile --only-binary=:all: --platform=manylinux2014_x86_64 --target=. -r ${abspath(path.module)}/scripts/lambda_function/requirements.txt",
":zip .",
]
}
]
}
This results in the following package getting built, even when running locally on macos platforms:
# module.lambda_function.local_file.archive_plan[0] must be replaced
-/+ resource "local_file" "archive_plan" {
~ content = jsonencode(
~ {
~ build_plan = [
- [
- "zip:embedded",
- "scripts/lambda_function",
- null,
],
- [
- "sh",
- "scripts/lambda_function",
- <<-EOT
cd `mktemp -d`
python3.9 -m pip install --no-compile --only-binary=:all: --platform=manylinux2014_x86_64 --target=. -r /Users/demo/tmp/test/scripts/lambda_function/requirements.txt
EOT,
],
- [
- "zip:embedded",
- ".",
- null,
],
+ [
+ "pip",
+ "python3.9",
+ "./scripts/lambda_function/requirements.txt",
+ null,
+ null,
],
+ [
+ "zip",
+ "./scripts/lambda_function",
+ null,
],
]
~ filename = "builds/9925cec5923cd3d1b3b77a4cd789ff95e4757db8b4d929b6f134b44ef2fe5d7a.zip" -> "builds/2576c817ebffdc69f333b98308abe4ce0389bb32513e67a9496393c2f0a3c5ae.zip"
# (2 unchanged elements hidden)
} # forces replacement
)
[...]
}
# module.lambda_function.null_resource.archive[0] must be replaced
[...]
module.lambda_function.null_resource.archive[0]: Creating...
module.lambda_function.null_resource.archive[0]: Provisioning with 'local-exec'...
module.lambda_function.null_resource.archive[0] (local-exec): Executing: ["python3" ".terraform/modules/lambda_function/package.py" "build" "--timestamp" "1663079193415791000" "builds/9925cec5923cd3d1b3b77a4cd789ff95e4757db8b4d929b6f134b44ef2fe5d7a.plan.json"]
[...]
module.lambda_function.null_resource.archive[0] (local-exec): Collecting cryptography
module.lambda_function.null_resource.archive[0] (local-exec): Using cached cryptography-38.0.1-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.1 MB)
Here a Lambda-ABI-compatible manylinux2014_x86_64 wheel for the cryptography package is chosen, but it should not be that difficult to get there.
Additional context
Here is how I use this module to compile cartography package on Mac M1 for Linux/x86_64:
module "lambda" {
# ...
source_path = jsonencode({
path = "verify_signature_handler.py"
pip_requirements = "requirements.txt"
patterns = <<END
!.*
.*\.py
.*\.so.*
END
})
build_in_docker = true
docker_pip_cache = true
}I just tested this exact code, and it worked with this module version 4.0.1 (latest) but should also work for the one you specified.
Also, for more complex build process, you can use Dockerfile as described in this comment.
--platform=linux/x86_64 public.ecr.aws/lambda/python:3.9 image is of course by default compatible with the lambda runtime). But in our case this would require docker (pip install) - in - docker (CI), which would open another can of worms.
I should have been more clear in the README and features in this module as soon as AWS added support for deploying Docker images into Lambda. This module is doing a bit too much when it comes to building and packaging. Adding even more features to it won't make it simpler to maintain. :)
I am going to close this issue as resolved.
I'm going to lock this issue because it has been closed for 30 days