Service discovery

Service discovery extracts all the potentially useful information from different sources, converts it to the configurations and exports them to the different destinations.

Pipeline

The service discovery pipeline has four jobs:

Job Description
discovery Dynamically discovers monitoring targets by collecting events from kubernetes API server. It collects POD and SERVICE events.
tag Dynamically add tags to discovered monitoring targets. Based on the POD and SERVICE fields and using patterns on them, one or more tags are attached to the monitoring targets.
build Dynamically creates data collection configurations for the monitored targets, using templates.
export Dynamically exports data collection configurations to allow netdata data collection plugins to use them. Data collection jobs in netdata are created and destroyed as needed.

Routing in a job and between jobs based on tags and selector.

Pipeline configuration:

name: <name>
discovery: <discovery_config>
tag: <tag_config>
build: <build_config>
export: <export_config>

Tags and selectors

Tag, build and export jobs have selector, the pipeline routes a target/config to the job only if its tags matches job selectors.

Both tags and selector are just lists of words.

A word must match the regex ^[a-zA-Z][a-zA-Z0-9=_.]*$.

Tags special cases:

  • -word: the word will be removed on tags merging.

Selectors special cases:

  • !word: should not contain the word.
  • word|word|word: should contain any word.

Discovery

Discovery job dynamically discovers targets using one of the supported service-discovery mechanisms.

Supported mechanisms:

Discovery configuration:

k8s:
  - <kubernetes_discovery_config>

Kubernetes

Kubernetes discoverer retrieves targets from Kubernetes' REST API. It always stays synchronized with the cluster state.

Configuration options:

# Mandatory. Tags to add to all discovered targets.
tags: <tags>

# Mandatory. The Kubernetes role of entities that should be discovered.
role: <role>

# Optional. Discover only targets that exist on the same node as service-discovery.
# This option works only for 'pod' role and it requires MY_NODE_NAME env variable to be set.
local_mode: <boolean>

# Optional. If omitted, all namespaces are used.
namespaces:
  - <namespace>

One of the following role types can be configured to discover targets:

  • pod
  • service

Pod Role

The pod role discovers all pods and exposes their containers as targets. For each declared port of a container, it generates single target. If there is no declared port it generates one target with empty Port, PortName and PortProtocol fields.

Available pod target fields:

Name Type Value
TUID string Namespace_Name_ContName_PortProtocol_Port
Address string PodIP:Port
Namespace string pod.metadata.namespace
Name string pod.metadata.name
Annotations map[string]string pod.metadata.annotations
Labels map[string]string pod.metadata.labels
NodeName string pod.spec.nodeName
PodIP string pod.status.podIP
ControllerName string pod.OwnerReferences.Controller.Name
ControllerKind string pod.OwnerReferences.Controller.Kind
ContName string pod.spec.containers.name
Image string pod.spec.containers.image
Env map[string]string pod.spec.containers.env + pod.spec.containers.envFrom
Port string pod.spec.containers.ports.containerPort
PortName string pod.spec.containers.ports.name
PortProtocol string pod.spec.containers.ports.protocol

Service Role

The service role discovers a target for each service port for each service.

Available service target fields:

Name Type Value
TUID string Namespace_Name_PortProtocol_Port
Address string Name.Namespace.svc:Port
Namespace string svc.metadata.namespace
Name string svc.metadata.name
Annotations map[string]string svc.metadata.annotations
Labels map[string]string svc.metadata.labels
Port string pod.spec.containers.ports.containerPort
PortName string pod.spec.containers.ports.name
PortProtocol string pod.spec.containers.ports.protocol
ClusterIP string svc.spec.clusterIP
ExternalName string svc.spec.externalName
Type string svc.spec.ports.type

Tag

Tag job tags targets discovered by discovery job. Its purpose is service identification.

Configuration is a list of tag rules:

- <tag_rule_config>

Tag rule configuration options:

# Mandatory. Routes targets to this tag rule with tags matching this selector.
selector: <selector>

# Mandatory. Tags to merge with the target tags if at least on of the match rules matches.
tags: <tags>

# Mandatory. Match rules, at least one should be defined. 
match:
  # Optional. Routes targets to this match rule with tags matching this selector.
  - selector: <selector>

    # Mandatory. Tags to merge with the target tags if this rule expression evaluates to true.
    tags: <tags>

    # Mandatory. Match expression.
    expr: <expression>

Match expression evaluation result should be true or false.

Expression syntax is go-template.

Available functions

Custom functions:

  • glob reports whether arg1 matches the shell file name pattern.
  • re reports whether arg1 contains any match of the regular expression pattern.

All these functions accepts two or more arguments, returning in effect:

func(arg1, arg2) || func(arg1, arg3) || func(arg1, arg4) ...

Build

Build job creates configurations from targets.

Configuration is a list of build rules:

- <build_rule_config>

Build rule configuration options:

# Mandatory. Routes targets to this rule with tags matching this selector.
selector: <selector>

# Mandatory. Tags to add to all built by this rule configurations.
tags: <tags>

# Mandatory. Apply rules, at least one should be defined. 
apply:
  # Mandatory. Routes targets to this apply rule with tags matching this selector.
  - selector: <selector>

    # Optional. Tags to add to configurations built by this apply rule.
    tags: <tags>

    # Mandatory. Configuration template.
    template: <template>

Template syntax is go-template.

Available functions

Custom functions:

  • glob reports whether arg1 matches the shell file name pattern.
  • re reports whether arg1 contains any match of the regular expression pattern.

All these functions accepts two or more arguments, returning in effect:

func(arg1, arg2) || func(arg1, arg3) || func(arg1, arg4) ...

Export

Export job exports configurations built by build job.

Supported exporters:

  • file

Export configuration:

file:
  - <file_exporter_config>

File

File exporter writes configurations to a specific file.

# Mandatory. Routes configurations to this exporter with tags matching this selector.
selector: <selector>

# Mandatory. Absolute path to a file.
filename: <filename>

Troubleshooting

Service-discovery has debug mode and special stdout exporter which is enabled only when it's running from the terminal.

CLI:

Usage:
  sd [OPTION]...

Application Options:
      --config-file= Configuration file path
      --config-map=  Configuration ConfigMap (name:key)
  -d, --debug        Debug mode

Help Options:
  -h, --help         Show this help message