AWS MSK and AWS Glue Schema Registry Configuration is Wrong
emircankilinc opened this issue · 0 comments
Hi team, we have separate accounts for MSK and AWS Glue Schema Registry, and using protocol buffer message format.
Prior to configuring the KafkaProducer, we perform assume role requests for the Glue Schema Registry because it needs another role for auth.
When initializing the KafkaProducer with the provided configuration properties, we encounter a given error.
Config map for KafkaProducer:
public Map<String, Object> mapToKafkaProducerConfig() {
Map<String, Object> conf = new HashMap<>();
conf.put(ProducerConfig.ACKS_CONFIG, getAcksConfig());
conf.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, getMskBootstrapServers());
conf.put(SaslConfigs.SASL_JAAS_CONFIG, "software.amazon.msk.auth.iam.IAMLoginModule required;");
conf.put(SaslConfigs.SASL_MECHANISM, "AWS_MSK_IAM");
conf.put(SaslConfigs.SASL_CLIENT_CALLBACK_HANDLER_CLASS, "software.amazon.msk.auth.iam.IAMClientCallbackHandler");
conf.put(KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
conf.put(VALUE_SERIALIZER_CLASS_CONFIG, GlueSchemaRegistryKafkaSerializer.class.getName());
conf.put(AWSSchemaRegistryConstants.DATA_FORMAT, DataFormat.PROTOBUF.name());
conf.put(AWSSchemaRegistryConstants.AWS_REGION, "eu-central-1");
conf.put(AWSSchemaRegistryConstants.REGISTRY_NAME,"registry-name");
conf.put(AWSSchemaRegistryConstants.SCHEMA_NAME,"schema-name");
conf.put(AWSSchemaRegistryConstants.PROTOBUF_MESSAGE_TYPE, ProtobufMessageType.POJO.getName());
return conf;
}
Error :
Caused by: software.amazon.awssdk.services.glue.model.AccessDeniedException: User: arn:aws:sts::000000000000:assumed-role/service-role/aws-sdk-java-000000000000 is not authorized to perform: glue:GetSchemaByDefinition on resource: arn:aws:glue:eu-central-1:000000000000:registry/registry-name because no identity-based policy allows the glue:GetSchemaByDefinition action (Service: Glue, Status Code: 400, Request ID: 00000x0x-x00x-00x0-0xx0-000000000x00)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
It seems like it is still trying to use MSK role to access Glue. How do we make it stop?
Performing the assume role request before setting the configuration properties for the Kafka producer.
try (StsClient stsClient = StsClient.builder().region(Region.EU_CENTRAL_1).build()) {
AssumeRoleRequest roleRequest = AssumeRoleRequest.builder()
.roleArn("arn:aws:iam::000000000000:role/glue-access")
.roleSessionName("kafka-consumer-cross-account-glue-schemaregistry-demo")
.build();
stsClient.assumeRole(roleRequest);
What part of the configuration is incorrect, and how can we properly set up the configuration for Glue and MSK? Can you provide guidance on resolving this issue?