Remove "time" from the "fields" sent to splunk_hec endpoints
matthewmodestino opened this issue · 1 comments
Is your feature request related to a problem? Please describe.
I am doing some testing on the fields and formats (event and raw) that the containerd logging pipeline sends to splunk hec.
When I send the event format to the raw endpoint, i can inspect what OTel is sending:
In the payload we can see the "indexed fields" we send. In those fields we should avoid sending massive cardinality for Splunk to index, like "time".
I believe we spoke about this in the past, and may have done it for docker, but im using containerd rendered logging pipeline from the chart.
Describe the solution you'd like
remove timestamps from fields sent by default to splunk to avoid indexing fields with super high cardinality.
its already being correctly set at the root time
field. It is also in the raw event
field if the user ever wants to extract it at search time.
Describe alternatives you've considered
I can workaround it by trying to drop this field downstream in splunk. It requires configuring a props/transforms to "null out" the indexed field value (i think).
Will also try dropping via config override on otel agent side.
Additional context
we should check all logging pipelines to ensure we are checking the fields we send for auto indexed extractions to avoid indexing/stroage impact, where it makes sense
This makes sense. Looking at the code, we export this as a field because it is listed as an attribute of the log record or an attribute of the resource log it is a part of, so this comes from the parsing.