"Cannot load native module 'Crypto.Hash._SHA256'" within Lambda Python3.7

Question

"Cannot load native module 'Crypto.Hash._SHA256'" within Lambda Python3.7

Closed this issue 5 years ago · 2 comments

Describe the bug
When a lambda is invoked (which contains the staxapp code), we get the above error message

To Reproduce
Using this link, create the Hello World app with python3.7

hello_world/app.py

import json

import requests
from staxapp.auth import StaxAuth
from staxapp.config import Config as StaxConfig

def lambda_handler(event, context):
     return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "hello world",
        }),
    }

hello_world/requirements.txt

requests
staxapp

Expected behavior
Build Completes
Screenshots

╰─➤  sam build --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
No Parameters detected in the template
2 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
No Parameters detected in the template
Building function 'HelloWorldFunction'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /Users/richardkeit/stax-test/staxapp/hello_world/requirements.txt --dest /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u
Full dependency closure: {pycryptodome==3.3.1(sdist), attrs==20.3.0(wheel), future==0.18.2(sdist), setuptools==50.3.2(wheel), python-dateutil==2.8.1(wheel), pycrypto==2.6.1(sdist), importlib-metadata==2.0.0(wheel), openapi-spec-validator==0.2.9(wheel), botocore==1.19.13(wheel), pyrsistent==0.17.3(sdist), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), ecdsa==0.16.0(wheel), pyyaml==5.3.1(sdist), envs==1.3(wheel), jmespath==0.10.0(wheel), urllib3==1.25.11(wheel), pyjwt==1.7.1(wheel), staxapp==1.0.5(wheel), idna==2.10(wheel), warrant==0.6.1(wheel), certifi==2020.11.8(wheel), zipp==3.4.0(wheel), python-jose-cryptodome==1.3.2(wheel), jsonschema==3.2.0(wheel), aws-requests-auth==0.4.3(wheel), semver==2.13.0(wheel), requests==2.24.0(wheel), chardet==3.0.4(wheel), prance==0.19.0(wheel), six==1.15.0(wheel)}
initial compatible: {attrs==20.3.0(wheel), setuptools==50.3.2(wheel), python-dateutil==2.8.1(wheel), importlib-metadata==2.0.0(wheel), openapi-spec-validator==0.2.9(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), ecdsa==0.16.0(wheel), envs==1.3(wheel), jmespath==0.10.0(wheel), urllib3==1.25.11(wheel), pyjwt==1.7.1(wheel), staxapp==1.0.5(wheel), idna==2.10(wheel), warrant==0.6.1(wheel), certifi==2020.11.8(wheel), zipp==3.4.0(wheel), python-jose-cryptodome==1.3.2(wheel), jsonschema==3.2.0(wheel), aws-requests-auth==0.4.3(wheel), semver==2.13.0(wheel), requests==2.24.0(wheel), chardet==3.0.4(wheel), prance==0.19.0(wheel), six==1.15.0(wheel)}
initial incompatible: {pyrsistent==0.17.3(sdist), pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist), future==0.18.2(sdist), pycrypto==2.6.1(sdist)}
Downloading missing wheels: {pyrsistent==0.17.3(sdist), pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist), future==0.18.2(sdist), pycrypto==2.6.1(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u pyrsistent==0.17.3
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u pycryptodome==3.3.1
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u pyyaml==5.3.1
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u future==0.18.2
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u pycrypto==2.6.1
compatible wheels after second download pass: {attrs==20.3.0(wheel), setuptools==50.3.2(wheel), python-dateutil==2.8.1(wheel), importlib-metadata==2.0.0(wheel), openapi-spec-validator==0.2.9(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), ecdsa==0.16.0(wheel), envs==1.3(wheel), jmespath==0.10.0(wheel), urllib3==1.25.11(wheel), pyjwt==1.7.1(wheel), staxapp==1.0.5(wheel), idna==2.10(wheel), warrant==0.6.1(wheel), certifi==2020.11.8(wheel), zipp==3.4.0(wheel), python-jose-cryptodome==1.3.2(wheel), jsonschema==3.2.0(wheel), aws-requests-auth==0.4.3(wheel), semver==2.13.0(wheel), requests==2.24.0(wheel), chardet==3.0.4(wheel), prance==0.19.0(wheel), six==1.15.0(wheel)}
Build missing wheels from sdists (C compiling True): {pyrsistent==0.17.3(sdist), pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist), future==0.18.2(sdist), pycrypto==2.6.1(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/pyrsistent-0.17.3.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/pycryptodome-3.3.1.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/PyYAML-5.3.1.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/future-0.18.2.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/pycrypto-2.6.1.tar.gz
compatible after building wheels (no C compiling): {attrs==20.3.0(wheel), future==0.18.2(wheel), setuptools==50.3.2(wheel), python-dateutil==2.8.1(wheel), importlib-metadata==2.0.0(wheel), openapi-spec-validator==0.2.9(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), ecdsa==0.16.0(wheel), envs==1.3(wheel), jmespath==0.10.0(wheel), urllib3==1.25.11(wheel), pyjwt==1.7.1(wheel), staxapp==1.0.5(wheel), idna==2.10(wheel), warrant==0.6.1(wheel), certifi==2020.11.8(wheel), zipp==3.4.0(wheel), python-jose-cryptodome==1.3.2(wheel), jsonschema==3.2.0(wheel), aws-requests-auth==0.4.3(wheel), semver==2.13.0(wheel), requests==2.24.0(wheel), chardet==3.0.4(wheel), prance==0.19.0(wheel), six==1.15.0(wheel)}
Build missing wheels from sdists (C compiling False): {pycrypto==2.6.1(sdist), pyrsistent==0.17.3(sdist), pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/pycrypto-2.6.1.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/pyrsistent-0.17.3.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/pycryptodome-3.3.1.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u /var/folders/qm/kl69fk2n73nb565tbtj02vl40000gp/T/tmpen4xy16u/PyYAML-5.3.1.tar.gz
compatible after building wheels (C compiling): {attrs==20.3.0(wheel), future==0.18.2(wheel), setuptools==50.3.2(wheel), python-dateutil==2.8.1(wheel), importlib-metadata==2.0.0(wheel), openapi-spec-validator==0.2.9(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), ecdsa==0.16.0(wheel), envs==1.3(wheel), jmespath==0.10.0(wheel), urllib3==1.25.11(wheel), pyjwt==1.7.1(wheel), staxapp==1.0.5(wheel), idna==2.10(wheel), warrant==0.6.1(wheel), certifi==2020.11.8(wheel), zipp==3.4.0(wheel), python-jose-cryptodome==1.3.2(wheel), jsonschema==3.2.0(wheel), aws-requests-auth==0.4.3(wheel), semver==2.13.0(wheel), requests==2.24.0(wheel), chardet==3.0.4(wheel), prance==0.19.0(wheel), six==1.15.0(wheel)}
Final compatible: {attrs==20.3.0(wheel), future==0.18.2(wheel), setuptools==50.3.2(wheel), envs==1.3(wheel), jmespath==0.10.0(wheel), urllib3==1.25.11(wheel), python-dateutil==2.8.1(wheel), pyjwt==1.7.1(wheel), importlib-metadata==2.0.0(wheel), staxapp==1.0.5(wheel), idna==2.10(wheel), warrant==0.6.1(wheel), openapi-spec-validator==0.2.9(wheel), certifi==2020.11.8(wheel), botocore==1.19.13(wheel), zipp==3.4.0(wheel), python-jose-cryptodome==1.3.2(wheel), jsonschema==3.2.0(wheel), aws-requests-auth==0.4.3(wheel), semver==2.13.0(wheel), boto3==1.16.13(wheel), requests==2.24.0(wheel), chardet==3.0.4(wheel), s3transfer==0.3.3(wheel), prance==0.19.0(wheel), ecdsa==0.16.0(wheel), six==1.15.0(wheel)}
Final incompatible: {pycrypto==2.6.1(wheel), pyrsistent==0.17.3(wheel), pycryptodome==3.3.1(wheel), pyyaml==5.3.1(wheel)}
Final missing wheels: {pycrypto==2.6.1(sdist), pyrsistent==0.17.3(sdist), pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/1.2.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
    requirements_path=self.manifest_path,
  File "/usr/local/Cellar/aws-sam-cli/1.2.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/Cellar/aws-sam-cli/1.2.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 207, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {pycrypto==2.6.1(sdist), pyrsistent==0.17.3(sdist), pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/1.2.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/usr/local/Cellar/aws-sam-cli/1.2.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 45, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {pycrypto==2.6.1(sdist), pyrsistent==0.17.3(sdist), pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist)}

Build Failed
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 72315, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': '1265b8a4-1fb3-4688-884b-ae960d28b9bf', 'installationId': '392697cc-c5eb-4319-b188-0acc2352d3a8', 'sessionId': '0c6c0be9-1171-405a-af2a-b87b91f7084b', 'executionEnvironment': 'CLI', 'pyversion': '3.7.9', 'samcliVersion': '1.2.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - {pycrypto==2.6.1(sdist), pyrsistent==0.17.3(sdist), pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist)}

Environment Details (please complete the following information):

OS:

╰─➤  uname -a
Darwin 192-168-1-198.tpgi.com.au 18.7.0 Darwin Kernel Version 18.7.0: Sun Dec  1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64

Python Version 3.7

╰─➤  sam --version                                                                                                                                                                                                                                                                                                                   2 ↵
SAM CLI, version 1.2.0

Additional context
The use of the SAM packaging approach is to simply allow for the easiest reproducible environment.
I am currently using the traditional lambda packaging approach described here

As suggested by @Anton0 , using a docker container mitigates this dep issue:

╰─➤  sam build --debug --use-container
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Starting Build inside a container
No Parameters detected in the template
2 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
No Parameters detected in the template
Building function 'HelloWorldFunction'

Fetching amazon/aws-sam-cli-build-image-python3.7 Docker container image............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Mounting /Users/richardkeit/stax-test/staxapp/hello_world as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch
Full dependency closure: {aws-requests-auth==0.4.3(wheel), idna==2.10(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), pyyaml==5.3.1(sdist), importlib-metadata==2.0.0(wheel), pycrypto==2.6.1(sdist), future==0.18.2(sdist), envs==1.3(wheel), pycryptodome==3.3.1(sdist), chardet==3.0.4(wheel), zipp==3.4.0(wheel), setuptools==50.3.2(wheel), prance==0.19.0(wheel), staxapp==1.0.5(wheel), warrant==0.6.1(wheel), ecdsa==0.16.0(wheel), urllib3==1.25.11(wheel), attrs==20.3.0(wheel), pyrsistent==0.17.3(sdist), python-jose-cryptodome==1.3.2(wheel), certifi==2020.11.8(wheel), six==1.15.0(wheel), jmespath==0.10.0(wheel), python-dateutil==2.8.1(wheel), openapi-spec-validator==0.2.9(wheel), pyjwt==1.7.1(wheel), requests==2.24.0(wheel), semver==2.13.0(wheel), jsonschema==3.2.0(wheel)}
initial compatible: {aws-requests-auth==0.4.3(wheel), idna==2.10(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), importlib-metadata==2.0.0(wheel), envs==1.3(wheel), chardet==3.0.4(wheel), zipp==3.4.0(wheel), setuptools==50.3.2(wheel), prance==0.19.0(wheel), staxapp==1.0.5(wheel), warrant==0.6.1(wheel), ecdsa==0.16.0(wheel), urllib3==1.25.11(wheel), attrs==20.3.0(wheel), python-jose-cryptodome==1.3.2(wheel), certifi==2020.11.8(wheel), six==1.15.0(wheel), jmespath==0.10.0(wheel), python-dateutil==2.8.1(wheel), openapi-spec-validator==0.2.9(wheel), pyjwt==1.7.1(wheel), requests==2.24.0(wheel), semver==2.13.0(wheel), jsonschema==3.2.0(wheel)}
initial incompatible: {pyyaml==5.3.1(sdist), pyrsistent==0.17.3(sdist), pycrypto==2.6.1(sdist), future==0.18.2(sdist), pycryptodome==3.3.1(sdist)}
Downloading missing wheels: {pyyaml==5.3.1(sdist), pyrsistent==0.17.3(sdist), pycrypto==2.6.1(sdist), future==0.18.2(sdist), pycryptodome==3.3.1(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /tmp/samcli/scratch pyyaml==5.3.1
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /tmp/samcli/scratch pyrsistent==0.17.3
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /tmp/samcli/scratch pycrypto==2.6.1
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /tmp/samcli/scratch future==0.18.2
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp37m --dest /tmp/samcli/scratch pycryptodome==3.3.1
compatible wheels after second download pass: {aws-requests-auth==0.4.3(wheel), idna==2.10(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), importlib-metadata==2.0.0(wheel), envs==1.3(wheel), chardet==3.0.4(wheel), zipp==3.4.0(wheel), setuptools==50.3.2(wheel), prance==0.19.0(wheel), staxapp==1.0.5(wheel), warrant==0.6.1(wheel), ecdsa==0.16.0(wheel), urllib3==1.25.11(wheel), attrs==20.3.0(wheel), python-jose-cryptodome==1.3.2(wheel), certifi==2020.11.8(wheel), six==1.15.0(wheel), jmespath==0.10.0(wheel), python-dateutil==2.8.1(wheel), openapi-spec-validator==0.2.9(wheel), pyjwt==1.7.1(wheel), requests==2.24.0(wheel), semver==2.13.0(wheel), jsonschema==3.2.0(wheel)}
Build missing wheels from sdists (C compiling True): {pycryptodome==3.3.1(sdist), pyyaml==5.3.1(sdist), pyrsistent==0.17.3(sdist), pycrypto==2.6.1(sdist), future==0.18.2(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/pycryptodome-3.3.1.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/PyYAML-5.3.1.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/pyrsistent-0.17.3.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/pycrypto-2.6.1.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/future-0.18.2.tar.gz
compatible after building wheels (no C compiling): {aws-requests-auth==0.4.3(wheel), idna==2.10(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), pyyaml==5.3.1(wheel), importlib-metadata==2.0.0(wheel), pycrypto==2.6.1(wheel), future==0.18.2(wheel), envs==1.3(wheel), pycryptodome==3.3.1(wheel), chardet==3.0.4(wheel), zipp==3.4.0(wheel), setuptools==50.3.2(wheel), prance==0.19.0(wheel), staxapp==1.0.5(wheel), warrant==0.6.1(wheel), ecdsa==0.16.0(wheel), urllib3==1.25.11(wheel), attrs==20.3.0(wheel), pyrsistent==0.17.3(wheel), python-jose-cryptodome==1.3.2(wheel), certifi==2020.11.8(wheel), six==1.15.0(wheel), jmespath==0.10.0(wheel), python-dateutil==2.8.1(wheel), openapi-spec-validator==0.2.9(wheel), pyjwt==1.7.1(wheel), requests==2.24.0(wheel), semver==2.13.0(wheel), jsonschema==3.2.0(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {aws-requests-auth==0.4.3(wheel), idna==2.10(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), pyyaml==5.3.1(wheel), importlib-metadata==2.0.0(wheel), pycrypto==2.6.1(wheel), future==0.18.2(wheel), envs==1.3(wheel), pycryptodome==3.3.1(wheel), chardet==3.0.4(wheel), zipp==3.4.0(wheel), setuptools==50.3.2(wheel), prance==0.19.0(wheel), staxapp==1.0.5(wheel), warrant==0.6.1(wheel), ecdsa==0.16.0(wheel), urllib3==1.25.11(wheel), attrs==20.3.0(wheel), pyrsistent==0.17.3(wheel), python-jose-cryptodome==1.3.2(wheel), certifi==2020.11.8(wheel), six==1.15.0(wheel), jmespath==0.10.0(wheel), python-dateutil==2.8.1(wheel), openapi-spec-validator==0.2.9(wheel), pyjwt==1.7.1(wheel), requests==2.24.0(wheel), semver==2.13.0(wheel), jsonschema==3.2.0(wheel)}
Final compatible: {aws-requests-auth==0.4.3(wheel), warrant==0.6.1(wheel), ecdsa==0.16.0(wheel), urllib3==1.25.11(wheel), idna==2.10(wheel), botocore==1.19.13(wheel), boto3==1.16.13(wheel), s3transfer==0.3.3(wheel), pyyaml==5.3.1(wheel), importlib-metadata==2.0.0(wheel), attrs==20.3.0(wheel), pyrsistent==0.17.3(wheel), pycrypto==2.6.1(wheel), python-jose-cryptodome==1.3.2(wheel), future==0.18.2(wheel), envs==1.3(wheel), certifi==2020.11.8(wheel), six==1.15.0(wheel), pycryptodome==3.3.1(wheel), chardet==3.0.4(wheel), zipp==3.4.0(wheel), jmespath==0.10.0(wheel), python-dateutil==2.8.1(wheel), setuptools==50.3.2(wheel), openapi-spec-validator==0.2.9(wheel), pyjwt==1.7.1(wheel), requests==2.24.0(wheel), prance==0.19.0(wheel), staxapp==1.0.5(wheel), semver==2.13.0(wheel), jsonschema==3.2.0(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
Build inside container was successful. Copying artifacts from container to host
Copying from container: /tmp/samcli/artifacts/. -> /Users/richardkeit/stax-test/staxapp/.aws-sam/build/HelloWorldFunction
Build inside container succeeded

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Answer 1 · 2020-11-24T04:24:31.000Z

After speaking to Richard I poked around this issue a bit and discovered that pycrypto probably shouldn't be being imported at all. The only use of the crypto package that I could find (and @Anton0 had the same result) was in the warrant dependency, however requirements.txt for this dependency state that it uses pycryptodome, which is a fork of pycrypto. Given that pycrypto was apparently abandoned in 2013, I think this is probably an error in the stax lib's requirements.txt and that pycrypto should actually be pycryptodome. This may or may not solve the original bug reported here but probably needs to be remedied either way.

Answer 2 · 2020-11-30T05:24:33.000Z

I'll try and break down what I think is happening, starting with the original issue.

The title of the issue appears to reference a runtime error and the logs posted seem to be build time (using sam) which threw me a little, but both of them I believe are the result of python needing to compile wheels for packages that don't provide their own against a specific OS, we're then getting stuck with packages like pycrypto where the resulting build (perhaps due to its use of C extensions) is incompatible between operating systems (i.e. an OSX built pycrypto doesn't work on a Linux Lambda.

Using SAM, you're getting the error earlier since it knows it needs a Linux package and can't find or build one with what you've given it - that's why the --use-container flag exists.

If your functions depend on packages that have natively compiled dependencies, use this flag to build your function inside an AWS Lambda-like Docker container.

If you don't use sam and do something like pip install staxapp -t dist/ then you won't get an error since it'll happily build a package for your running OS (say OSX), but when you upload and run the code in Lambda, you'll get the error you're seeing.

Cannot load native module 'Crypto.Hash._SHA256'" within Lambda Python3.7

So it's the same problem - just depending on the tool you'll know about it earlier (build time) or later (runtime).

The other complicating factor is that as per the log originally provided it's more than one package it's:

{pycrypto==2.6.1(wheel), pyrsistent==0.17.3(wheel), pycryptodome==3.3.1(wheel), pyyaml==5.3.1(wheel)}

To address the pickup from @jamesinc we do appear to be installing pycrypto unnecessarily, I've raised a PR to remove that from the next release, and it's a great catch.

However this only resolves the problem with 1/4 of the packages, the other three including pycryptodome require builds against an OS that's compatible with Lambda, whether or not OSX is I'm not sure (I'll try and test that theory out).

Unfortunately, after looking more at this, I'm not sure what we can do to address it from our side short of removing those dependencies all together.

I can't work around the fact that building a package for one OS/Arch may not work on another.

You can see a similar (the same) issues raised against sam-cli and the recommendation is consistantly --use-container which I think may be the most prudent situation (or if you aren't using sam-cli ensuring your build environment is compatible with Lambda).