Software for monitoring of individual processes on Linux with Prometheus.
This repository is the home of the process-exporter, a prometheus exporter to monitor individual processes on Linux.
The exporter takes two parameters to select and de-select which processes to monitor.
-log.format value
Set the log target and format. Example: "logger:syslog?appname=bob&local=7" or "logger:stdout?json=true" (default "logger:stderr")
-log.level value
Only log messages with the given severity or above. Valid levels: [debug, info, warn, error, fatal]
-process.nowatch string
Processes to (no)watch
You should provided at least one process to watch.
The parameter process.watch should be a comma-seperated list of regular expressions of processes to watch
The parameter process.nowatch is a filter that removes processes from the list provided by process.watch
-process.watch string
Processes to (no)watch
You should provided at least one process to watch.
The parameter process.watch should be a comma-seperated list of regular expressions of processes to watch
The parameter process.nowatch is a filter that removes processes from the list provided by process.watch
-version
Print version information.
-web.listen-address string
Address to listen on for web interface and telemetry. (default ":8980")
-web.telemetry-path string
Path under which to expose metrics. (default "/metrics")
The selection of the processes to return is rather simple.
- Of every process found, the command line is taken.
- The contents of the command line is matched with the arguments of process.watch. No match, go to next proccess.
- Next, match the command line with the arguments of process.nowatch, Match, go to next process
- If process should be watch, add a record for this process, labeling it with the argument from process.watch and the pid found.
This way of selecting has some side effects:
- the process of the exporter itself, has all arguments of process.watch in its command line
- If you run 'curl http://localhost:8980/metrics | grep <process>' where process is in the process.watch list, you will get an extra record in the output for each metric, since 'grep <process>' actually matches.
It's up to the user to add enough arguments to process.nowatch to make sure only the required processes are being reporter. It's advised to at least give process-expoter as an argument to process.nowatch.
If you want to run the process-exporter inside docker, you need to make sure the process-exporter has access to the process information of the host. This can be done by mounting the /proc filesystem to the docker container, and setting an environment variable to point the process-exporter to the right directory.
Docker example:
docker run --rm -d -v /:/rootfs -e HOST_PROC=/rootfs/proc -e HOST_SYS=/rootfs/sys -e HOST_ETC=/rootfs/etc -p 8980:8980 --name=process-exporter rberwald/process-exporter /process-exporter -process.watch process1,process2 -process.nowatch process-exporter
sum by (process) (irate(indivdual_process{instance="<server>:8980", metric=~"cpu_.*"}[1m]))
indivdual_process{instance="<server>:8980", metric=~"memory_.*", process="<process>"}
The process-exporter itself is not using any environment variables.
The library used to get the statistics of the processes (https://github.com/shirou/gopsutil/tree/master/process) is using these environment variables:
- HOST_PROC
- HOST_SYS
- HOST_ETC
Setting these variables is necessary when running the process-exporter inside a docker containers.
Overview of the metrics exporter by this exporter.
indivdual_process{metric="cpu_guest",pid="915",process="<process>"} indivdual_process{metric="cpu_guestNice",pid="915",process="<process>"} indivdual_process{metric="cpu_idle",pid="915",process="<process>"} indivdual_process{metric="cpu_iowait",pid="915",process="<process>"} indivdual_process{metric="cpu_irq",pid="915",process="<process>"} indivdual_process{metric="cpu_nice",pid="915",process="<process>"} indivdual_process{metric="cpu_softirq",pid="915",process="<process>"} indivdual_process{metric="cpu_steal",pid="915",process="<process>"} indivdual_process{metric="cpu_stolen",pid="915",process="<process>"} indivdual_process{metric="cpu_system",pid="915",process="<process>"} indivdual_process{metric="cpu_user",pid="915",process="<process>"} indivdual_process{metric="memory_data",pid="915",process="<process>"} indivdual_process{metric="memory_dirty",pid="915",process="<process>"} indivdual_process{metric="memory_lib",pid="915",process="<process>"} indivdual_process{metric="memory_rss",pid="915",process="<process>"} indivdual_process{metric="memory_shared",pid="915",process="<process>"} indivdual_process{metric="memory_text",pid="915",process="<process>"} indivdual_process{metric="memory_vms",pid="915",process="<process>"}
Where process is the name as given in the arguments process.watch.
There are some support scripts in the scripts/
sub-directory. The scripts:
build
: Script to compile the GO code in thegolang
container and build a Docker container with the binary.run
: Script to quickly run the result ofbuild
on a local docker instance.
List of things to do in the future versions:
- Make the selection of processes smarter
- Solve all inline ToDo remarks
- Add automated tests
- Add automated build