databricks/iceberg-kafka-connect

Access kafka cluster topic in confluent cloud

Opened this issue · 2 comments

I'm trying to connect this connector to the kafka topic present in my confluent cloud. I was able to access the confluent kafka topic from an ec2 instance. Now I'm creating this connector in same VPC, subnet and attaching same security group as of ec2 instance but I'm unable to get the data.
These are the configurations that I'm using but I'm getting timeout

iceberg.kafka.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<CONFLUENT_API_KEY>" password="<CONFLUENT_API_SECRET>";
iceberg.kafka.security.protocol=SASL_SSL
iceberg.kafka.sasl.mechanism=PLAIN

connector.class=io.tabular.iceberg.connect.IcebergSinkConnector
table.write-format=parquet
iceberg.tables.evolve-schema-enabled=true
iceberg.fs.s3a.path.style.access=true
table.namespace=pocckafkaflink2
iceberg.catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
tasks.max=1
topics=confluent_test_topic2
iceberg.catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
iceberg.catalog.client.region=us-east-1
iceberg.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
iceberg.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
iceberg.tables=pocckafkaflink2.mskmytopicmaib
value.converter.schemas.enable=false
iceberg.catalog.warehouse=s3://bucket/data4
value.converter=org.apache.kafka.connect.json.JsonConverter
table.auto-create=true 


is this the correct format to provide sasl.jaas.config, security.protocol and sasl.mechanism

@arshahmad1 Hi, Muhammad, have you found the fix for this problem ?