Unable to Change Kafka Topic Name in Serverless Trigger

[mahesh-atidealo]

Are you certain it's a bug?

• Yes, it looks like a bug

Is the issue caused by a plugin?

• It is not a plugin issue

Are you using the latest v3 release?

• Yes, I'm using the latest v3 release

Is there an existing issue for this?

• I have searched existing issues, it hasn't been reported yet

Issue description

When attempting to modify the Kafka topic name associated with a serverless trigger in our environment, we encountered a critical issue where the operation fails with an error message indicating that the consumer group ID already exists.

Failing use-case: We are using kafka topic "my-sample-v1" and migrating to "my-sample-v2" while consumer group remains same

Error Message:
CREATE_FAILED: ProcessorEventSourceMappingKafkaMySampleTopicv2 (AWS::Lambda::EventSourceMapping) Resource handler returned message: "An event source mapping with event source ("kafka-brokers:port"), selfManagedKafkaEventSourceConfig.consumerGroupId ("MySample-Lambda") already exists. Please update or delete the existing mapping with UUID abcd36ef-g166-k263-11b0-00013f812145 (Service: Lambda, Status Code: 409, Request ID: 12345678-0501-1235-1234-12345690ec0d)" (RequestToken: 12345678-0001-1111-2222-12345690ec0d, HandlerErrorCode: AlreadyExists)

Programming Language: Java 17

Additional Notes:

• This problem occurs consistently across multiple attempts and environments, indicating a potential bug in the serverless platform's handling of Kafka trigger topic name change keeping the consumer group same.
• We also tried deployment after manually deactivating the the kafka trigger, however seems to be no errors but change is not applied as topic remains same
• As a workaround, we tried manually removing the trigger and redeploying with new topic name and it worked.

Service configuration (serverless.yml) content

service: MySampleLambda
frameworkVersion: '3'
configValidationMode: error
provider:
  name: aws
  stage: test
  deploymentMethod: direct
  runtime: java17
  region: aws-region
package:
  artifact: target/MySampleLambda.jar

functions:
  processor:
    handler: com.sample.handler.MySampleEventHandler
    events:
      - kafka:
          topic: my-sample-v2
          consumerGroupId: MySampleLambda
          batchSize: 1
          bootstrapServers:
            - kafka-brokers:port
          accessConfigurations: All working access configurations

Command name and used flags

N/A

Command output

N/A

Environment information

linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1