Jenkins Remoting monitoring with OpenTelemetry Plugin

Introduction
Getting started
Configuration options
Specification
Contributing
LICENSE
Links

Introduction

Goal

The goal of this project:

collect telemetry data(metrics, traces, logs) of remoting module with OpenTelemetry.
send the telemetry data to OpenTelemetry Protocol endpoint

Which OpenTelemetry endpoint to use and how to visualize the data are up to users. Collect telemetry data of Jenkins Remoting using OpenTelemetry.

OpenTelemetry

An observability framework for cloud-native software

OpenTelemetry is a collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data(metrics, logs, and traces) for analysis in order to understand your software’s performance and behavior.

Getting started

1. Setup OpenTelemetry protocol endpoint and monitoring backends

We prepare docker-compose.yaml to set up them. Use it if you just want to try.

Clone repository from https://github.com/jenkinsci/remoting-opentelemetry-plugin.

$ cd example
$ docker-compose up

This will set up

OpenTelemetry Collector
Loki for Log aggregation
Prometheus for metric backend
Grafana for log and metric visualization
- datasource is already configured

2. Download monitoring-engine

Download remoting-opentelemetry-engine-<tag>.jar from Release Page.

We will use this JAR as java agent when launching agent.

3. Create `logging.properties` file.

Use io.jenkins.plugins.remotingopentelemetry.engine.log.OpenTelemetryLogHandler for handler.

logging.properties

handlers=io.jenkins.plugins.remotingopentelemetry.engine.log.OpenTelemetryLogHandler,java.util.logging.ConsoleHandler
.level=INFO

4. Launch Jenkins agent

Setup jenkins controller and launch agent with -javaagent and -loggingConfig option.

$ export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:55680
$ java \
-javaagent remoting-opentelemetry-engine.jar \
-jar agent.jar \
-jnlpUrl <jnlp url> \
-loggingConfig logging.properties

5. Explore logs and metrics

Open Grafana: http://localhost:3000/explore

Configuration options

We can configure the monitoring engine via environment variables.

environment variable	require	example / description
OTEL_EXPORTER_OTLP_ENDPOINT	true	`http://localhost:55680`
OTEL_EXPORTER_OTLP_ENDPOINT	true	Target to which the exporter is going to send spans, metrics or logs.
SERVICE_INSTANCE_ID	false	90caeb02-a5ba-4827-bb3e-63babecfa893
SERVICE_INSTANCE_ID	false	The string ID of the service instance. If not provided, UUID will be generated and used every time the agent launches.
REMOTING_OTEL_METRIC_FILTER	false	"system\.cpu\..*"
REMOTING_OTEL_METRIC_FILTER	false	Set regex filter for metrics. The metrics whose name match the regex will be collected. The default value is ".*" and collect all the metrics.

Specification

Resource

Following resource attributes will be provided.

key	value	description
service_namespace	"jenkins"	This value will be configurable in the future.
service_namespace	"jenkins-agent"	This value will be configurable in the future.
service_instance_id	Node name

Logs

Only logs emitted via java.util.logging will be collected for now.

Following attributes will be provided.

key	example	description
log.level	INFO	Log level name. See `java.util.logging.Level.getName`.
code.namespace	hudson.remoting.jnlp.Main$CuiListener	The name of the class that (allegedly) issued the logging request.
code.function	status	The name of the method that (allegedly) issued the logging request.
exception.type	java.io.IOException	The class name of the throwable associated with the log record.
exception.message	Broken pipe	The detail message string of the throwable associated with the log record.
exception.stacktrace	java.io.IOException: Broken pipe at hudson.remoting.Engine.innerRun(Engine.java:784) at hudson.remoting.Engine.run(Engine.java:575)	The stacktrace the throwable associated with the log record.

Spans

TBD

Metrics

Following metrics will be collected.

metrics	unit	label key	label value	description
system.cpu.load	1			System CPU load. See `com.sun.management.OperatingSystemMXBean.getSystemCpuLoad`
system.cpu.load.average.1m				System CPU load average 1 minute See `java.lang.management.OperatingSystemMXBean.getSystemLoadAverage`
system.memory.usage	byte	state	`used`, `free`	see `com.sun.management.OperatingSystemMXBean.getTotalPhysicalMemorySize` and `com.sun.management.OperatingSystemMXBean.getFreePhysicalMemorySize`
system.memory.utilization	1			System memory utilization, see `com.sun.management.OperatingSystemMXBean.getTotalPhysicalMemorySize` and `com.sun.management.OperatingSystemMXBean.getFreePhysicalMemorySize`. Report 0% if no physical memory is discovered by the JVM.
system.paging.usage	byte	state	`used`, `free`	see `com.sun.management.OperatingSystemMXBean.getFreeSwapSpaceSize` and `com.sun.management.OperatingSystemMXBean.getTotalSwapSpaceSize`.
system.paging.utilization	1			see `com.sun.management.OperatingSystemMXBean.getFreeSwapSpaceSize` and `com.sun.management.OperatingSystemMXBean.getTotalSwapSpaceSize`. Report 0% if no swap memory is discovered by the JVM.
system.filesystem.usage	byte	device	(identifier)	System level filesystem usage. Linux only (get mount data from /proc/mounts).
		state	`used`, `free`
		type	`ext4`, `tmpfs`, etc.
		mode	`rw`,`ro`,etc.
		mountpoint	(path)
system.filesystem.utilization	1	device	(identifier)	System level filesystem utilization (0.0 to 1.0). Linux only (get mount data from /proc/mounts).
		state	`used`, `free`
		type	`ext4`, `tmpfs`, etc.
		mode	`rw`,`ro`,etc.
		mountpoint	(path)
process.cpu.load	%			Process CPU load. See `com.sun.management.OperatingSystemMXBean.getProcessCpuLoad`.
process.cpu.time	ns			Process CPU time. See `com.sun.management.OperatingSystemMXBean.getProcessCpuTime`.
runtime.jvm.memory.area	bytes	type	`used`, `committed`, `max`	see MemoryUsage
runtime.jvm.memory.area	bytes	area	`heap`, `non_heap`	see MemoryUsage
runtime.jvm.memory.pool	bytes	type	`used`, `committed`, `max`	see MemoryUsage
runtime.jvm.memory.pool	bytes	pool	`PS Eden Space`, `G1 Old Gen`…	see MemoryUsage
runtime.jvm.gc.time	ms	gc	`G1 Young Generation`, `G1 Old Generation`, …	see GarbageCollectorMXBean
runtime.jvm.gc.count	1	gc	`G1 Young Generation`, `G1 Old Generation`, …	see GarbageCollectorMXBean

Contributing

Refer to our contribution guidelines.

LICENSE

Licensed under MIT, see LICENSE

Aki-7/remoting-opentelemetry-plugin