splunk.custom.header option does not seem to work

[guilhemmarchand]

Hi,

It looks like the splunk.custom.header does work as it should, we are trying to use the feature as follows:

Example of a message containing a custom header:

Key (1 bytes): 1
Value (395 bytes): {"timestamp":"2020-09-15 15:09:03.947", "VSAD_ID":"acme_amer", "eventKey":"1", "uuid":"c2cc7db8-3a12-4d3e-bff0-cf33a7927862", "message":"nximmnoanlzkpwktwtynuabuemvzivtjsmiuhwwuscwvqvkfioflpqjfyhafpzbsuskeuwnyblqtdctupcqnqkmardpdrbghftkibvtpnvveildpzuftdmusfkgsytuvaydnckanileergykrvfvscukijybsledarqbwizdsoztypawzahtwrxseajgswozjzvffveihvsnsusrdrjtzvypucppydgdildmpvqmrzhtpkmjntyckdslhhgxoide"}
Headers: header_index=kafka_connect_demo_amer,header_host=kafka.west.amer,header_source=kafka:west:amer,header_sourcetype=kafka:app:xxxxx,company=ACME
Partition: 0	Offset: 139
--
% Reached end of topic kafka_app_acme_amer [0] at offset 140

The header includes a customer header called "company" with a value of "ACME"

The sink connector configuration is the following:

{
  "name": "kafka-sink-kafka-demo-west:amer",
  "config": {
    "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector",
    "splunk.hec.raw": "true",
    "tasks.max": "6",
    "topics": "kafka_app_acme_amer",
    "splunk.indexes": "main",
    "splunk.sources": "kafka:west:undefined",
    "splunk.header.custom": "company",
    "splunk.hec.uri": "http://10.0.0.234:8088",
    "splunk.header.sourcetype": "header_sourcetype",
    "splunk.hec.ssl.validate.certs": "false",
    "splunk.sourcetypes": "kafka:app:undefined",
    "splunk.header.source": "header_source",
    "name": "kafka-sink-kafka-demo-west:amer",
    "splunk.header.support": "true",
    "splunk.hec.raw.line.breaker": "####",
    "splunk.header.host": "header_host",
    "splunk.hec.token": "3584634f-a663-49cf-a2fa-23521bb515aa",
    "splunk.header.index": "header_index"
  },
  "tasks": [
    {
      "connector": "kafka-sink-kafka-demo-west:amer",
      "task": 0
    },
    {
      "connector": "kafka-sink-kafka-demo-west:amer",
      "task": 1
    },
    {
      "connector": "kafka-sink-kafka-demo-west:amer",
      "task": 2
    },
    {
      "connector": "kafka-sink-kafka-demo-west:amer",
      "task": 3
    },
    {
      "connector": "kafka-sink-kafka-demo-west:amer",
      "task": 4
    },
    {
      "connector": "kafka-sink-kafka-demo-west:amer",
      "task": 5
    }
  ],
  "type": "sink"
}

The event is indexed properly, however there are no signs at all of the custom field :

Splunk OOTB Metadata work ok (index, source, sourcetype, host), however it looks like something got broken for the custom header support.

Thank you in advance

Guilhem

[guilhemmarchand]

Root cause:

This option will only work with the event endpoint, this is not compatible with the raw endpoint.
The documentation needs to be updated to reflect that fact properly, as currently nothing specifies this is the case

When running against the raw endpoint, Metadata are defined within the URL, which is working differently with the event endpoint.

Fix required in the docs ;-)

[cheneysplunk]

Thanks @guilhemmarchand. Filed a doc Jira ticket https://jira.splunk.com/browse/ADDON-29376. Will work with doc writers.