aws/aws-sdk-java

Amazon S3 v2 Has Behavioural Difference When Use KMS Encryption

abilhanli opened this issue · 1 comments

abilhanli commented

Upcoming End-of-Support

  • I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.

Describe the bug

Essentially, we are going to move to AWS JAVA SDK Version 2 from V1, due to end of support.
Based on migration guide https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/s3-encryption-migration.html It is suggested to do followings:

1. Update existing clients to read new formats. Update your application to use version 1.11.837 or later of the AWS SDK for Java and redeploy the application. This enables the Amazon S3 client-side encryption service clients in your application to decrypt objects created by V2 service clients. If your application uses multiple AWS SDKs, you must update each SDK separately.
2. Migrate encryption and decryption clients to V2. Once all of your V1 encryption clients can read V2 encryption formats, update the Amazon S3 client-side encryption and decryption clients in your application code to use their V2 equivalents.

We upgraded our dependencies to latest 1.+.

    api 'com.amazonaws:aws-java-sdk-s3:1.12.747'
    api 'com.amazonaws:aws-java-sdk-core:1.12.747'
    api 'com.amazonaws:aws-java-sdk-kms:1.12.747'

After we use V2 objects from V1 jar, we noticed that an extra validation introduced in V2 which is to validate KMS materials.
Because of the validation an checked exception thrown with a message like Provided encryption materials do not match information retrieved from the encrypted object by AWS API which we can not handle in our service, taking consideration the client builder obj is used and all those validations are hidden in the API.

As a result, our integration test is failing. If we simply fix our integration tests and ignore it, we might end up a customer impact.

We kindly ask make this new validation optional.

Expected Behavior

We do not have the Exception explained in the bug, when using V1 objects.
This suggests that V2 is not backward compatible due to the additional validation for encryption materials.

We expect that V1 behaviour is kept in V2.

Could you please make this validation optional to make it backward compatible?

Current Behavior

After using the version 2 objects, we had following exception

Caused by: java.lang.IllegalStateException: Provided encryption materials do not match information retrieved from the encrypted object
    at com.amazonaws.services.s3.internal.crypto.v2.ContentCryptoMaterial.validateMaterialsForDecrypt(ContentCryptoMaterial.java:857)
    at com.amazonaws.services.s3.internal.crypto.v2.ContentCryptoMaterial.fromObjectMetadata0(ContentCryptoMaterial.java:390)
    at com.amazonaws.services.s3.internal.crypto.v2.ContentCryptoMaterial.fromObjectMetadata(ContentCryptoMaterial.java:342)
    at com.amazonaws.services.s3.internal.crypto.v2.S3CryptoModuleAE.decipherWithMetadata(S3CryptoModuleAE.java:263)
    at com.amazonaws.services.s3.internal.crypto.v2.S3CryptoModuleAE.decipher(S3CryptoModuleAE.java:158)
    at com.amazonaws.services.s3.internal.crypto.v2.S3CryptoModuleAE.getObjectSecurely(S3CryptoModuleAE.java:136)
    at com.amazonaws.services.s3.AmazonS3EncryptionClientV2.getObject(AmazonS3EncryptionClientV2.java:241)
    at

Reproduction Steps

In order to reproduce to same exception, you can simply provide a kms id which is different than what is used in encrypted obj and then call getObject method on the client.

The code block to create an encrypted client object using V2 could be like

    protected AWSKMS getAWSKMS(AWSCredentials awsCredentials, String KMSRegion) {
        return AWSKMSClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
                .withRegion(KMSRegion) //String value
                .build();
    }

protected AmazonS3EncryptionClientV2Builder getAmazonS3EncryptionClientBuilderV2(AWSCredentials awsCredentials, String KMSRegion, String kmsKeyId) {
        AWSKMS kmsClient = getAWSKMS(awsCredentials, KMSRegion);

        return AmazonS3EncryptionClientV2Builder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
                .withKmsClient(kmsClient)
                .withCryptoConfiguration(
                        new CryptoConfigurationV2().withCryptoMode(CryptoMode.AuthenticatedEncryption).withRangeGetMode(CryptoRangeGetMode.ALL))
                .withEncryptionMaterialsProvider(new KMSEncryptionMaterialsProvider(kmsKeyId));
    }

and then call the client like

s3Client.getObject(new GetObjectRequest(bucket, objectName));

Possible Solution

Make it optional to be backward compatible.

Additional Information/Context

No response

AWS Java SDK version used

1.12.747(Version 2 objects based on migration guide)

JDK version used

11

Operating System and version

Mac Osx 14.5 (23F79)

debora-ito commented

Hi @abilhanli

There's a little confusion when it comes to the v1 encryption client: we indeed have a AmazonS3EncryptionClientV2, but in this case both v1 and v2 Encryption Clients are in the com.amazonaws groupId, so it means both are in the end-of-support path.

The equivalent of the Encryption Client in Java SDK v2 is the standalone Amazon S3 Encryption Client 3.x:

    <groupId>software.amazon.encryption.s3</groupId>
    <artifactId>amazon-s3-encryption-client-java</artifactId>
    <version>3.x</version>

The S3 Encryption Client 3.x is maintained by the Encryption Tools team, and the project has its own Github repo: https://github.com/aws/amazon-s3-encryption-client-java. There's some Migration to 3.x tips in the Developer Guide here. I suggest you try the 3.x encryption client and see if the issue still persists, and if it does please open an issue in the new repo.

For more info on Java SDK v1 to v2 migration, check the Migrate to version 2 section of the AWS SDK for Java Dev Guide.