MSK AWS connect
Opened this issue · 9 comments
Im trying to set the connector in AWS connect with a MSK serverless cluster in AWS but I keep getting the error:
[Worker-016e5f49a1715bc84] [2023-05-10 17:40:33,607] WARN [iceberg-sink-connector\|task-0] Unable to retrieve the requested metadata. (software.amazon.awssdk.regions.internal.util.EC2MetadataUtils:398) -- [Worker-016e5f49a1715bc84] [2023-05-10 17:40:33,608] ERROR [iceberg-sink-connector\|task-0] WorkerSinkTask{id=iceberg-sink-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:191) [Worker-016e5f49a1715bc84] software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@c649ef2: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@52410e10: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@2d28feba: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@1c19d17a: Unable to retrieve region information from EC2 Metadata service. Please make sure the application is running on EC2.]
Any suggestion on how to declare region in the connector?
this is my config by the way:
"connector.class"= "com.getindata.kafka.connect.iceberg.sink.IcebergSink"
"tasks.max"= "1"
"topics"= "sqa-sdiv-data-warehouse.public.test_2,sqa-sdiv-data-warehouse.public.test_table"
"upsert"= true
"upsert.keep-deletes"= true
"table.auto-create"= true
"table.write-format"= "parquet"
"table.namespace"= "sdiv"
"table.prefix"= "cdc_"
"iceberg.catalog-impl"= "org.apache.iceberg.aws.glue.GlueCatalog"
"iceberg.fs.defaultFS"= "s3a://sqa-subdiv-kafka-iceberg-poc/sdiv-iceberg"
"iceberg.warehouse"= "s3a://sqa-subdiv-kafka-iceberg-poc/sdiv-iceberg"
"iceberg.fs.s3a.endpoint.region" = "us-west-2"
"iceberg.fs.s3a.endpoint" = "s3.us-west-2.amazonaws.com"
"iceberg.com.amazonaws.services.s3.enableV4"= true
"iceberg.com.amazonaws.services.s3a.enableV4"= true
"iceberg.fs.s3a.aws.credentials.provider"= "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
"iceberg.fs.s3a.path.style.access"= true
"iceberg.fs.s3a.impl"= "org.apache.hadoop.fs.s3a.S3AFileSystem"
I am having the exact same issue. Unable to specify the region to the worker in any way
Facing the same issue.
Solved by modifying the connector and compiling it again, tomorrow will share the modified code so you can do the same
Ok, it will be very helpful. waiting for updated connector.
@clazalde , I would appreciate if you can share the updated connector. Thanks in advance.
In the public void start(Map<String, String> properties) {
Add the following:
System.setProperty("aws.region", "us-west-2");
Replace it with your region
@clazalde , I have set the aws.region. Now I am getting below error:-
Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. Error: Unable to execute HTTP request: connect timed out (org.apache.kafka.connect.runtime.WorkerSinkTask:612)
I am also facing the same issue while trying to use this connector in AWS MSK cluster. Any input would be greatly appreciated. 🙏
software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain