spring-cloud/spring-cloud-stream-binder-aws-kinesis

4.x shouldn't reuse the Dynamodb tables created by version 2.x

asinbow opened this issue · 4 comments

asinbow commented

In what version(s) of Spring Cloud Stream Binder for AWS Kinesis are you seeing this issue?

4.0.0

Describe the bug

Where upgrading spring-cloud-stream-binder-aws-kinesis from 2.2.0 to 4.0.0, saw the error in production

"Got an exception software.amazon.awssdk.services.dynamodb.model.DynamoDbException:
The provided key element does not match the schema (Service: DynamoDb, Status Code: 400, Request ID: ***) during 
[ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null',
timestamp=null, stream='quoting', shard='shardId-000000000000', reset=false}, state=NEW}] task invocation.
Process will be retried on the next iteration."

To Reproduce
Reuse the Dynamodb tables, SpringIntegrationMetadataStore and SpringIntegrationLockRegistry, created by version 2.x.

Expected behavior

N/A

Sample

N/A

asinbow commented

Analysis

By inspecting the schema of in production and in my local dev

// $ aws dynamodb describe-table --region us-west-1 --table-name SpringIntegrationMetadataStore 
{
    "Table": {
        "AttributeDefinitions": [
            {
                "AttributeName": "KEY", // previous schema in production
                "AttributeType": "S"
            }
        ],
        "TableName": "SpringIntegrationMetadataStore",
        "KeySchema": [
            {
                "AttributeName": "KEY",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "ACTIVE",
        "CreationDateTime": "2023-03-31T15:24:51.433000+08:00",
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 0,
            "WriteCapacityUnits": 0
        },
        "TableSizeBytes": 304,
        "ItemCount": 2,
        "TableArn": "arn:aws:dynamodb:us-west-1:250343255435:table/SpringIntegrationMetadataStore",
        "TableId": "37b874da-58db-4def-89ec-b111e2072130",
        "BillingModeSummary": {
            "BillingMode": "PAY_PER_REQUEST",
            "LastUpdateToPayPerRequestDateTime": "2023-03-31T15:24:51.433000+08:00"
        },
        "DeletionProtectionEnabled": false
    }
}
// $ awslocal dynamodb describe-table --table-name SpringIntegrationMetadataStore
{
    "Table": {
        "AttributeDefinitions": [
            {
                "AttributeName": "metadataKey", // new schema in my local dev
                "AttributeType": "S"
            }
        ],
        "TableName": "SpringIntegrationMetadataStore",
        "KeySchema": [
            {
                "AttributeName": "metadataKey",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "ACTIVE",
        "CreationDateTime": 1690504059.416,
        "ProvisionedThroughput": {
            "LastIncreaseDateTime": 0.0,
            "LastDecreaseDateTime": 0.0,
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 0,
            "WriteCapacityUnits": 0
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:us-east-1:000000000000:table/SpringIntegrationMetadataStore",
        "TableId": "df9cb0ad-a08a-4444-b758-f94a433a1586",
        "BillingModeSummary": {
            "BillingMode": "PAY_PER_REQUEST",
            "LastUpdateToPayPerRequestDateTime": 1690504059.416
        },
        "Replicas": []
    }
}

There is definitely a breaking change in
https://github.com/spring-projects/spring-integration-aws/blob/a1d2b0d3a86e0862a0be7021957e61e5b0a8d7c6/src/main/java/org/springframework/integration/aws/metadata/DynamoDbMetadataStore.java#L70-L78

asinbow commented

The solution is to create new DynamoDB tables and use them, e.g.

spring:
  cloud:
    stream:
       kinesis:
         binder:
           checkpoint:
             table: SpringIntegrationAwsV3MetadataStore
           locks:
             table: SpringIntegrationAwsV3LockRegistry
asinbow commented

Just to record the issue. Feel free to close it.

BTW it's the only breaking change for us, all the rest work well during the upgrade. Thanks!

artembilan commented

Yes, it was expected breaking change after migrating to AWS SDK v2.
Sorry for inconvenience: you can migrate those old items to a new table manually or just ignore them altogether since they are just some metadata supporting Kinesis input bindings which is OK to have as fresh in your fresh application instance.
There is another breaking change which is addressed via: #192.

Yes, I'm closing this one since there is nothing what can be done from the framework perspective.

  • 👍1