aws/aws-dynamodb-encryption-python

downstream tests in upstream library failing but upstream tests passing

mattsb42-aws opened this issue · 1 comments

mattsb42-aws commented

As identified in pyca/cryptography#4979, some recent changes to the integration test suite setup caused pyca/cryptography's downstream tests to start failing[1].

This should have been caught in our upstream tests, but it was not.

There are two issues here:

  1. Our upstream tests are passing through environment variable configurations, which is masking the errors.
  2. The integration test parameterization setup is calling functions that require that environment variables are set during test collection, when those functions should only be called during test execution.

#1 is simple enough to fix, and that revealed #2[2].

The root of this is that while fixtures (such as integration_test_utils.cmk_arn) are executed when the test is executed, pytest_generate_tests is executed at test collection time.

[1] https://travis-ci.org/pyca/cryptography/jobs/579259777
[2]

============================================================================================ test session starts =============================================================================================
platform darwin -- Python 2.7.13, pytest-4.6.5, py-1.8.0, pluggy-0.12.0
cachedir: .tox/test-upstream-requirements-py27/.pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/bullocm/git/aws-dynamodb-encryption-python/.hypothesis/examples')
rootdir: /Users/bullocm/git/aws-dynamodb-encryption-python, inifile: setup.cfg
plugins: hypothesis-4.34.0, cov-2.7.1, forked-1.0.2, mock-1.10.4, xdist-1.29.0
collected 8679 items / 4 errors / 7417 deselected / 1258 selected                                                                                                                                            

=================================================================================================== ERRORS ===================================================================================================
_________________________________________________________________________ ERROR collecting test/integration/encrypted/test_client.py _________________________________________________________________________
test/integration/encrypted/test_client.py:29: in pytest_generate_tests
    set_parameterized_kms_cmps(metafunc)
test/integration/integration_test_utils.py:52: in set_parameterized_kms_cmps
    inner_cmp = AwsKmsCryptographicMaterialsProvider(key_id=cmk_arn_value())
test/integration/integration_test_utils.py:37: in cmk_arn_value
    AWS_KMS_KEY_ID
E   ValueError: Environment variable "AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID" must be set to a valid KMS CMK ARN for integration tests to run
________________________________________________________________________ ERROR collecting test/integration/encrypted/test_resource.py ________________________________________________________________________
test/integration/encrypted/test_resource.py:29: in pytest_generate_tests
    set_parameterized_kms_cmps(metafunc)
test/integration/integration_test_utils.py:52: in set_parameterized_kms_cmps
    inner_cmp = AwsKmsCryptographicMaterialsProvider(key_id=cmk_arn_value())
test/integration/integration_test_utils.py:37: in cmk_arn_value
    AWS_KMS_KEY_ID
E   ValueError: Environment variable "AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID" must be set to a valid KMS CMK ARN for integration tests to run
_________________________________________________________________________ ERROR collecting test/integration/encrypted/test_table.py __________________________________________________________________________
test/integration/encrypted/test_table.py:29: in pytest_generate_tests
    set_parameterized_kms_cmps(metafunc)
test/integration/integration_test_utils.py:52: in set_parameterized_kms_cmps
    inner_cmp = AwsKmsCryptographicMaterialsProvider(key_id=cmk_arn_value())
test/integration/integration_test_utils.py:37: in cmk_arn_value
    AWS_KMS_KEY_ID
E   ValueError: Environment variable "AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID" must be set to a valid KMS CMK ARN for integration tests to run
____________________________________________________________________ ERROR collecting test/integration/material_providers/test_aws_kms.py ____________________________________________________________________
test/integration/material_providers/test_aws_kms.py:36: in pytest_generate_tests
    set_parameterized_kms_cmps(metafunc, require_attributes=False)
test/integration/integration_test_utils.py:52: in set_parameterized_kms_cmps
    inner_cmp = AwsKmsCryptographicMaterialsProvider(key_id=cmk_arn_value())
test/integration/integration_test_utils.py:37: in cmk_arn_value
    AWS_KMS_KEY_ID
E   ValueError: Environment variable "AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID" must be set to a valid KMS CMK ARN for integration tests to run
mattsb42-aws commented

The simplest solution to this would be for the parameterization to emit a callable that builds the CMP rather than emitting the CMP already built.

This will require some modification in our tests, but it will at least put the execution at the right time.

An alternate option that I considered was to only check that the value is not None in the cmk_arn fixture, but that would create inconsistent behavior and could cause tests to fail in unexpected ways if integration tests are run when not configured.