CP 7.2.2: Kafka Connect connectors don't seem to respect SSL settings for value converter

[evakkuri]

Hi! I'm trying to create a secure connection between Kafka Connect and Confluent Schema Registry, but I'm having issues.

Both Schema Registry and Kafka Connect are running in Kubernetes, with the following container images:
Kafka Connect: based on confluentinc/cp-kafka-connect:7.2.2, added Datagen Source and JDBC connector jars, no other changes
Schema Registry: confluentinc/cp-schema-registry:7.2.2

I have configured Kafka Connect with environment variables such that by running kubectl exec -n kafka <pod name> -- cat /etc/kafka-connect/kafka-connect.properties, I get the results in this gist. Note, for sharing, I have removed all password values and replaced other sensitive information with tokens like <...>.

I then try to post a new connector, for instance Datagen Source Connector as described in this gist. In the end I get an javax.net.ssl.SSLHandshakeException error with this stack trace.

What is weird is that if I look at Kafka Connect logging for creating the connector, it shows all properties related to Schema Registry SSL as null, as described in this gist. So, it looks like the connector does not get the settings for some reason.

You can see from the gists that I have tried setting the SSL-related values in many different ways, if that's the issue, then apologies. However, my understanding is that unknown settings are simply ignored by Kafka Connect.

I can get Schema Registry to reply to me with curl such that it responds with a certificate with canonical name like CN=<service name>-schema-registry, signed by the same certificate as /certs/truststore-clients/ca.crt. Therefore, the certs themselves should be fine. Also, all connections to my Kafka cluster work fine, no issue there.

Any idea what's the issue or if I'm doing something wrong? I'm happy to provide more details as needed.

[evakkuri]

Update: If I provide the trust store for the Schema Registry connection in the connector creation API call, then the trust store is configured properly:

In API call body:

...
"value.converter.schema.registry.ssl.truststore.type": "PEM",
"value.converter.schema.registry.ssl.truststore.location": "/certs/truststore-clients/ca.crt",
...

In Connect logging:

...
schema.registry.ssl.truststore.certificates = null
schema.registry.ssl.truststore.location = /certs/truststore-clients/ca.crt
schema.registry.ssl.truststore.password = null
schema.registry.ssl.truststore.type = PEM
...