issue to send data tosplunk , com.splunk.hecclient.HecException: No channels are available / registered with LoadBalancer

[Techqueries]

Hello Members,

I am trying to use kafka-connect-splunk to get JSON messages from few topics be inserted into Splunk.
I already have a kafka cluster on and running in shared environment, there is no issues with that. I created a kafka connector. And modified the connect-distributed.properties etc. (details below). I also configured the splunk as per the documentation. So far I am able to see the correct output of commands such as

{ "name": "kafka-connect-splunk",
"config": { "connector.class":"com.splunk.kafka.connect.SplunkSinkConnector",
"tasks.max": "1",
"topics": "syslog-itid-splunkBO",
"splunk.hec.uri": "https:localhost:8088",
"splunk.hec.token":"******" ,
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"org.apache.kafka.connect.storage.StringConverter",
"splunk.hec.ssl.validate.certs" : "false",
"splunk.hec.raw" : "true",
"splunk.hec.ack.enabled" : "false",
"splunk.hec.ack.poll.interval" : "20",
"splunk.hec.ack.poll.threads" : "2",
"splunk.hec.total.channels" : "2",
"splunk.hec.lb.poll.interval" : "200" } }

Splunk connector running without any issue but when try to send data to Splunk i am getting below issue

ERROR com.splunk.kafka.connect.SplunkSinkTask: failed to send batch b341f8d5-fc46-4911-b109-4ebf96dc78a0
com.splunk.hecclient.HecException: No channels are available / registered with LoadBalancer
at com.splunk.hecclient.LoadBalancer.send(LoadBalancer.java:99)
at com.splunk.hecclient.Hec.send(Hec.java:234)
at com.splunk.kafka.connect.SplunkSinkTask.send(SplunkSinkTask.java:291)
at com.splunk.kafka.connect.SplunkSinkTask.sendEvents(SplunkSinkTask.java:283)
at com.splunk.kafka.connect.SplunkSinkTask.handleEvent(SplunkSinkTask.java:255)
at com.splunk.kafka.connect.SplunkSinkTask.put(SplunkSinkTask.java:105)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:546)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:326)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:228)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

The message i sent to splunk is in json format

{"event":"splunk data ingetion,stamp: 2021-08-02T10:30:51","sourcetype":"syslog"}

i have disabled cert validation ( hec.ssl.validate.cert to false) in the config and tried as well but not worked.

Could you please advise how to make json messages make their way to splunk,
Please let me know if you need any other configuration properties in splunk side . Kafka connect is running without any issue but i am not able send the data to splunk

[luckyj5]

@Techqueries Can you try and enable below setting for json events?
splunk.hec.json.event.formatted - Set this setting true for events that are already in HEC format.
Valid options are true or false.
Thanks!

[Techqueries]

Its not worked after setting splunk.hec.json.event.formatted to true. But the issue is fixed due to the proxy issue , after removing the proxy the issue got resolved.