No httpclient implementations found on the context classloader
jchipmunk opened this issue · 6 comments
Hi,
I use kafka-kubernetes-config-provider v.1.0.0 deployed in the directory specified in the plugin.path worker configuration property. This version works great!
After updating to v.1.0.1, the Kafka Connect application startup failed with the following error:
[2022-08-15T14:08:43,248][ERROR][category=org.apache.kafka.connect.cli.ConnectDistributed] Stopping due to errorio.fabric8.kubernetes.client.KubernetesClientException: No httpclient implementations found on the context classloader, please ensure your classpath includes an implementation jar
at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:171)
at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:153)
at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:145)
at io.strimzi.kafka.AbstractKubernetesConfigProvider.configure(AbstractKubernetesConfigProvider.java:63)
at io.strimzi.kafka.KubernetesConfigMapConfigProvider.configure(KubernetesConfigMapConfigProvider.java:17)
at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:572)
at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:515)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:452)
at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:405)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
The reason for the error is that io.fabric8.kubernetes.client.utils.HttpClientUtils class loads HttpClient implementation using context classloader for current thread:
public static HttpClient createHttpClient(Config config) {
ServiceLoader<HttpClient.Factory> loader = ServiceLoader.load(HttpClient.Factory.class);
...
In this particular case the current thread is org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader class because org.apache.kafka.connect.runtime.isolation.PluginClassLoader class (who knows about libs of kafka-kubernetes-config-provider) is only used to instantiate the ConfigProvider. For more details, see org.apache.kafka.common.config.AbstractConfig class:
private Map<String, ConfigProvider> instantiateConfigProviders(Map<String, String> indirectConfigs, Map<String, ?> providerConfigProperties) {
...
ConfigProvider provider = Utils.newInstance(entry.getValue(), ConfigProvider.class); // PluginClassLoader is used here
provider.configure(configProperties); // DelegatingClassLoader is used here
...
How do you add it to the plugin directory? The libraries are int he ZIp file and they are in the dependency tree:
[INFO] +- io.strimzi:kafka-kubernetes-config-provider:jar:1.0.1:compile
[INFO] | \- io.fabric8:kubernetes-client:jar:6.0.0:compile
[INFO] | +- io.fabric8:kubernetes-client-api:jar:6.0.0:compile
[INFO] | | +- io.fabric8:kubernetes-model-core:jar:6.0.0:compile
[INFO] | | | \- io.fabric8:kubernetes-model-common:jar:6.0.0:compile
[INFO] | | +- io.fabric8:kubernetes-model-apps:jar:6.0.0:compile
[INFO] | | +- io.fabric8:kubernetes-model-batch:jar:6.0.0:compile
[INFO] | | +- io.fabric8:kubernetes-model-certificates:jar:6.0.0:compile
[INFO] | | \- io.fabric8:kubernetes-model-extensions:jar:6.0.0:compile
[INFO] | +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.0.0:runtime
[INFO] | | +- com.squareup.okhttp3:okhttp:jar:3.12.12:runtime
[INFO] | | | \- com.squareup.okio:okio:jar:1.15.0:runtime
[INFO] | | \- com.squareup.okhttp3:logging-interceptor:jar:3.12.12:runtime
[INFO] | \- io.fabric8:zjsonpatch:jar:0.3.0:compile
So whether you use the ZIP file or pull from Maven, it should be there without the need for any changes. I do not seem to have any problems with using it without adding / specifying the additional dependency.
How do you add it to the plugin directory? The libraries are int he ZIp file and they are in the dependency tree:
[INFO] +- io.strimzi:kafka-kubernetes-config-provider:jar:1.0.1:compile [INFO] | \- io.fabric8:kubernetes-client:jar:6.0.0:compile [INFO] | +- io.fabric8:kubernetes-client-api:jar:6.0.0:compile [INFO] | | +- io.fabric8:kubernetes-model-core:jar:6.0.0:compile [INFO] | | | \- io.fabric8:kubernetes-model-common:jar:6.0.0:compile [INFO] | | +- io.fabric8:kubernetes-model-apps:jar:6.0.0:compile [INFO] | | +- io.fabric8:kubernetes-model-batch:jar:6.0.0:compile [INFO] | | +- io.fabric8:kubernetes-model-certificates:jar:6.0.0:compile [INFO] | | \- io.fabric8:kubernetes-model-extensions:jar:6.0.0:compile [INFO] | +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.0.0:runtime [INFO] | | +- com.squareup.okhttp3:okhttp:jar:3.12.12:runtime [INFO] | | | \- com.squareup.okio:okio:jar:1.15.0:runtime [INFO] | | \- com.squareup.okhttp3:logging-interceptor:jar:3.12.12:runtime [INFO] | \- io.fabric8:zjsonpatch:jar:0.3.0:compileSo whether you use the ZIP file or pull from Maven, it should be there without the need for any changes. I do not seem to have any problems with using it without adding / specifying the additional dependency.
Some details of my configuration
I have the directory that is used for plugins (connectors, converters, transformations, config providers) and is configured accordingly in the connect-distributed.properties file:
plugin.path=/opt/kafka/plugins
The kafka-kubernetes-config-provider plugin with all the necessary libraries is located in this directory:
bash-5.1$ ls -la /opt/kafka/plugins/kafka-kubernetes-config-provider-1.0.1
total 9452
drwxr-xr-x 2 root root 4096 Aug 15 13:50 .
drwxr-xr-x 3 root root 50 Aug 15 13:50 ..
-rw-r--r-- 1 1001 116 75714 Aug 3 19:30 jackson-annotations-2.13.3.jar
-rw-r--r-- 1 1001 116 374895 Aug 3 19:30 jackson-core-2.13.3.jar
-rw-r--r-- 1 1001 116 1536542 Aug 3 19:30 jackson-databind-2.13.3.jar
-rw-r--r-- 1 1001 116 52020 Aug 3 19:30 jackson-dataformat-yaml-2.13.3.jar
-rw-r--r-- 1 1001 116 121201 Aug 3 19:30 jackson-datatype-jsr310-2.13.3.jar
-rw-r--r-- 1 1001 116 11054 Aug 3 19:30 kafka-kubernetes-config-provider-1.0.1.jar
-rw-r--r-- 1 1001 116 393908 Aug 3 19:30 kubernetes-client-6.0.0.jar
-rw-r--r-- 1 1001 116 405972 Aug 3 19:30 kubernetes-client-api-6.0.0.jar
-rw-r--r-- 1 1001 116 36110 Aug 3 19:30 kubernetes-httpclient-okhttp-6.0.0.jar
-rw-r--r-- 1 1001 116 444555 Aug 3 19:30 kubernetes-model-apps-6.0.0.jar
-rw-r--r-- 1 1001 116 248112 Aug 3 19:30 kubernetes-model-batch-6.0.0.jar
-rw-r--r-- 1 1001 116 145404 Aug 3 19:30 kubernetes-model-certificates-6.0.0.jar
-rw-r--r-- 1 1001 116 19677 Aug 3 19:30 kubernetes-model-common-6.0.0.jar
-rw-r--r-- 1 1001 116 4331115 Aug 3 19:30 kubernetes-model-core-6.0.0.jar
-rw-r--r-- 1 1001 116 544079 Aug 3 19:30 kubernetes-model-extensions-6.0.0.jar
-rw-r--r-- 1 1001 116 12483 Aug 3 19:30 logging-interceptor-3.12.12.jar
-rw-r--r-- 1 1001 116 427674 Aug 3 19:30 okhttp-3.12.12.jar
-rw-r--r-- 1 1001 116 88732 Aug 3 19:30 okio-1.15.0.jar
-rw-r--r-- 1 1001 116 331605 Aug 3 19:30 snakeyaml-1.30.jar
-rw-r--r-- 1 1001 116 35518 Aug 3 19:30 zjsonpatch-0.3.0.jar
i.e. the kubernetes-httpclient-okhttp library is also provided.
Regarding my explanation above, the DelegatingClassLoader loads any classes other than plugins (connectors, converters, transformations, config providers) using the application classloader (/opt/kafka/libs). So if you place the kafka-kubernetes-config-provider plugin with all the necessary libraries into Kafka's libs directory, then you will not have such a problem.
I also note that DefaultKubernetesClient class is now marked as deprecated.
Ok, I guess I get the point now. But it also makes it impossible to use it with the other than OkHttp implementations because you hardcode a single implementation I guess which might be a problem for other users.
It seemed to me that this is unlikely but if such user scenarios exist, then the problem should be fixed deeper at the kubernetes-client library, perhaps in the same way as it was done for KubernetesClientBuilder.
I will try to create the required request to the kubernetes-client repository.
TBH, I did not tried the other clients. One of the appeals would be that the Jetty client is already part of Kafka. So using that would decrease the footprint. But it depends on Jetty 11, so not sure it would work with Jetty 9 which is used by Kafka (and probably will be used until Kafka 4.0 when Java 8 is dropped). So I guess until then we can maybe proceed with your PR.