Service discovery

Service discovery extracts all the potentially useful information from different sources, converts it to the configurations and exports them to the different destinations.

Pipeline

The service discovery pipeline has four jobs:

Job	Description
discovery	Dynamically discovers monitoring targets by collecting events from kubernetes API server. It collects POD and SERVICE events.
tag	Dynamically add tags to discovered monitoring targets. Based on the POD and SERVICE fields and using patterns on them, one or more tags are attached to the monitoring targets.
build	Dynamically creates data collection configurations for the monitored targets, using templates.
export	Dynamically exports data collection configurations to allow netdata data collection plugins to use them. Data collection jobs in netdata are created and destroyed as needed.

Routing in a job and between jobs based on tags and selector.

Pipeline configuration:

name: <name>
discovery: <discovery_config>
tag: <tag_config>
build: <build_config>
export: <export_config>

Tags and selectors

Tag, build and export jobs have selector, the pipeline routes a target/config to the job only if its tags matches job selectors.

Both tags and selector are just lists of words.

A word must match the regex ^[a-zA-Z][a-zA-Z0-9=_.]*$.

Tags special cases:

-word: the word will be removed on tags merging.

Selectors special cases:

!word: should not contain the word.
word|word|word: should contain any word.

Discovery

Discovery job dynamically discovers targets using one of the supported service-discovery mechanisms.

Supported mechanisms:

kubernetes

Discovery configuration:

k8s:
  - <kubernetes_discovery_config>

Kubernetes

Kubernetes discoverer retrieves targets from Kubernetes' REST API. It always stays synchronized with the cluster state.

Configuration options:

# Mandatory. Tags to add to all discovered targets.
tags: <tags>

# Mandatory. The Kubernetes role of entities that should be discovered.
role: <role>

# Optional. Discover only targets that exist on the same node as service-discovery.
# This option works only for 'pod' role and it requires MY_NODE_NAME env variable to be set.
local_mode: <boolean>

# Optional. If omitted, all namespaces are used.
namespaces:
  - <namespace>

One of the following role types can be configured to discover targets:

pod
service

Pod Role

The pod role discovers all pods and exposes their containers as targets. For each declared port of a container, it generates single target. If there is no declared port it generates one target with empty Port, PortName and PortProtocol fields.

Available pod target fields:

Name	Type	Value
`TUID`	string	`Namespace_Name_ContName_PortProtocol_Port`
`Address`	string	`PodIP:Port`
`Namespace`	string	pod.metadata.namespace
`Name`	string	pod.metadata.name
`Annotations`	map[string]string	pod.metadata.annotations
`Labels`	map[string]string	pod.metadata.labels
`NodeName`	string	pod.spec.nodeName
`PodIP`	string	pod.status.podIP
`ControllerName`	string	pod.OwnerReferences.Controller.Name
`ControllerKind`	string	pod.OwnerReferences.Controller.Kind
`ContName`	string	pod.spec.containers.name
`Image`	string	pod.spec.containers.image
`Env`	map[string]string	pod.spec.containers.env + pod.spec.containers.envFrom
`Port`	string	pod.spec.containers.ports.containerPort
`PortName`	string	pod.spec.containers.ports.name
`PortProtocol`	string	pod.spec.containers.ports.protocol

Service Role

The service role discovers a target for each service port for each service.

Available service target fields:

Name	Type	Value
`TUID`	string	`Namespace_Name_PortProtocol_Port`
`Address`	string	`Name.Namespace.svc:Port`
`Namespace`	string	svc.metadata.namespace
`Name`	string	svc.metadata.name
`Annotations`	map[string]string	svc.metadata.annotations
`Labels`	map[string]string	svc.metadata.labels
`Port`	string	pod.spec.containers.ports.containerPort
`PortName`	string	pod.spec.containers.ports.name
`PortProtocol`	string	pod.spec.containers.ports.protocol
`ClusterIP`	string	svc.spec.clusterIP
`ExternalName`	string	svc.spec.externalName
`Type`	string	svc.spec.ports.type

Tag

Tag job tags targets discovered by discovery job. Its purpose is service identification.

Configuration is a list of tag rules:

- <tag_rule_config>

Tag rule configuration options:

# Mandatory. Routes targets to this tag rule with tags matching this selector.
selector: <selector>

# Mandatory. Tags to merge with the target tags if at least on of the match rules matches.
tags: <tags>

# Mandatory. Match rules, at least one should be defined. 
match:
  # Optional. Routes targets to this match rule with tags matching this selector.
  - selector: <selector>

    # Mandatory. Tags to merge with the target tags if this rule expression evaluates to true.
    tags: <tags>

    # Mandatory. Match expression.
    expr: <expression>

Match expression evaluation result should be true or false.

Expression syntax is go-template.

Available functions

go-template built-in functions.
sprig functions.
custom functions.

Custom functions:

glob reports whether arg1 matches the shell file name pattern.
re reports whether arg1 contains any match of the regular expression pattern.

All these functions accepts two or more arguments, returning in effect:

func(arg1, arg2) || func(arg1, arg3) || func(arg1, arg4) ...

Build

Build job creates configurations from targets.

Configuration is a list of build rules:

- <build_rule_config>

Build rule configuration options:

# Mandatory. Routes targets to this rule with tags matching this selector.
selector: <selector>

# Mandatory. Tags to add to all built by this rule configurations.
tags: <tags>

# Mandatory. Apply rules, at least one should be defined. 
apply:
  # Mandatory. Routes targets to this apply rule with tags matching this selector.
  - selector: <selector>

    # Optional. Tags to add to configurations built by this apply rule.
    tags: <tags>

    # Mandatory. Configuration template.
    template: <template>

Template syntax is go-template.

Available functions

go-template built-in functions.
sprig functions.
custom functions.

Custom functions:

glob reports whether arg1 matches the shell file name pattern.
re reports whether arg1 contains any match of the regular expression pattern.

All these functions accepts two or more arguments, returning in effect:

func(arg1, arg2) || func(arg1, arg3) || func(arg1, arg4) ...

Export

Export job exports configurations built by build job.

Supported exporters:

file

Export configuration:

file:
  - <file_exporter_config>

File

File exporter writes configurations to a specific file.

# Mandatory. Routes configurations to this exporter with tags matching this selector.
selector: <selector>

# Mandatory. Absolute path to a file.
filename: <filename>

Troubleshooting

Service-discovery has debug mode and special stdout exporter which is enabled only when it's running from the terminal.

CLI:

Usage:
  sd [OPTION]...

Application Options:
      --config-file= Configuration file path
      --config-map=  Configuration ConfigMap (name:key)
  -d, --debug        Debug mode

Help Options:
  -h, --help         Show this help message

ilyam8/agent-service-discovery

Service discovery

Pipeline

Tags and selectors

Discovery

Kubernetes

Pod Role

Service Role

Tag

Available functions

Build

Available functions

Export

File

Troubleshooting