Getting Started • Getting Involved • Migrating from Smart Agent

Components • Monitoring • Security • Sizing • Troubleshooting

Splunk OpenTelemetry Connector for Kubernetes

The Splunk OpenTelemetry Connector for Kubernetes is a Helm chart for the Splunk Distribution of OpenTelemetry Collector. This chart creates a Kubernetes DaemonSet along with other Kubernetes objects in a Kubernetes cluster and provides a unified way to receive, process and export metric, trace, and log data for Splunk Observability Cloud.

Installations that use this distribution can receive direct help from Splunk's support teams. Customers are free to use the core OpenTelemetry OSS components (several do!) and we will provide best effort guidance to them for any issues that crop up, however only the Splunk distributions are in scope for official Splunk support and support-related SLAs.

This distribution currently supports:

Splunk APM via the sapm exporter. The otlphttp exporter can be used with a custom configuration. More information available here.
Splunk Infrastructure Monitoring via the signalfx exporter. More information available here.
Splunk Log Observer via the splunk_hec exporter.
Splunk Cloud or Splunk Enterprise via the splunk_hec exporter.

🚧 This project is currently in BETA. It is officially supported by Splunk. However, breaking changes MAY be introduced.

Supported Kubernetes distributions

This helm chart is tested and works with default configurations on the following Kubernetes distributions:

While this helm chart should work for other Kubernetes distributions, it may require additional configurations applied to values.yaml.

Getting Started

Prerequisites

The following components required to use the helm chart:

How to install

To install splunk-otel-collector in k8s cluster at least three parameters must be provided:

splunkRealm (default us0): Splunk realm to send telemetry data to.
splunkAccessToken: Your Splunk org access token.
clusterName: arbitrary value that will identify your Kubernetes cluster in Splunk.

To deploy the chart run the following commands replacing the parameters above with their appropriate values.

$ helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart
$ helm install my-splunk-otel-collector --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster" splunk-otel-collector-chart/splunk-otel-collector

Instead of setting helm values as arguments a yaml file can be provided:

$ helm install my-splunk-otel-collector --values my_values.yaml splunk-otel-collector-chart/splunk-otel-collector

How to uninstall

To uninstall/delete a deployment with name my-splunk-otel-collector:

$ helm delete my-splunk-otel-collector

The command removes all the Kubernetes components associated with the chart and deletes the release.

Advanced Configuration

The values.yaml lists all supported configurable parameters for this chart, along with detailed explanation. Read through it to understand how to configure this chart.

Also check examples of chart configuration.

At the minimum you need to configure the following values.

clusterName: my-k8s-cluster
splunkAccessToken: xxxxxx
splunkRealm: us0

Cloud provider

Use the provider parameter to provide information about the cloud provider, if any.

aws - Amazon Web Services
gcp - Google Cloud
azure - Microsoft Azure

This value can be omitted if none of the values apply.

Kubernetes distribution

Use the distro parameter to provide information about underlying Kubernetes deployment. This parameter allows the connector to automatically scrape additional metadata. The supported options are:

eks - Amazon EKS
gke - Google GKE
aks - Azure AKS

This value can be omitted if none of the values apply.

Deployment environment

Optional environment parameter can be used to specify an additional deployment.environment attribute that will be added to all the telemetry data. It will help Splunk Observability users to investigate data coming from different source separately. Value examples: development, staging, production, etc.

environment: production

Disable particular types of telemetry

By default all telemetry data (metrics, traces and logs) is collected from the Kubernetes cluster. It's possible to disable any kind of telemetry with the following parameters:

metricsEnabled: false
tracesEnabled: false
logsEnabled: false

For example, to install the connector only for logs:

$ helm install my-splunk-otel-collector \
  --set="splunkRealm=us0,splunkAccessToken=xxxxxx,clusterName=my-cluster,metricsEnabled=false,tracesEnabled=false" \
  splunk-otel-collector-chart/splunk-otel-collector

Additional telemetry sources

Use autodetect config option to enable additional telemetry sources.

Set autodetect.prometheus=true if you want the otel-collector agent to scrape prometheus metrics from pods that have generic prometheus-style annotations:

prometheus.io/scrape: true: Prometheus metrics will be scraped only from pods having this annotation;
prometheus.io/path: path to scrape the metrics from, default /metrics;
prometheus.io/port: port to scrape the metrics from, default 9090.

Set autodetect.istio=true, if the otel-collector agent in running in Istio environment, to make sure that all traces, metrics and logs reported by Istio collected in a unified manner.

For example to enable both Prometheus and Istio telemetry add the following lines to your values.yaml file:

autodetect:
  istio: true
  prometheus: true

Pre-rendered Kubernetes resources

The rendered directory contains pre-rendered Kubernetes resource manifests.

Upgrade guidelines

0.26.4 to 0.27.0

#163 Auto-detection of prometheus metrics is disabled by default: If you rely on automatic prometheus endpoints detection to scrape prometheus metrics from pods in your k8s cluster, make sure to add this configuration to your values.yaml:

autodetect:
  prometheus: true

License

Apache Software License version 2.0.

owais/splunk-otel-collector-chart