Export metrics from Logstash to Prometheus. The project was created as rewrite of existing awesome application logstash_exporter, which was also written in Go, but it was not maintained for a long time. A lot of code was reused from the original project.
Important: Because of limited workforces, this project is tested only against a single Logstash version. You can check the tested version in docker-compose.yml file. Using this exporter with other versions of Logstash may not work properly (although most of the metrics should work).
The application can be run in two ways:
- using the binary executable
- using the Docker image
Additionally Helm chart is provided for easy deployment to Kubernetes.
The binary executable can be downloaded from the releases page.
Linux binary is available under https://github.com/kuskoman/logstash-exporter/releases/download/v${VERSION}/logstash-exporter-linux.
The binary can be run without additional arguments, as the configuration is loaded from the .env file and environment variables.
Each binary should contain a SHA256 checksum file, which can be used to verify the integrity of the binary.
VERSION="test-tag" \
OS="linux" \
wget "https://github.com/kuskoman/logstash-exporter/releases/download/${VERSION}/logstash-exporter-${OS}" && \
wget "https://github.com/kuskoman/logstash-exporter/releases/download/${VERSION}/logstash-exporter-${OS}.sha256" && \
sha256sum -c logstash-exporter-${OS}.sha256
It is recommended to use the binary executable in combination with the systemd service. The application should not require any of root privileges, so it is recommended to run it as a non-root user.
The unstable version of the application can be downloaded from the
GitHub Actions.
The latest successful build can be found under the Go application CI/CD workflow (already selected in the link).
To download the binary, simply go to the link location, click on the latest successful build, and download the binary
from the Artifacts section on the bottom of the page.
You are able to download artifact from any workflow run, not only master branch. To do that, go to
GitHub Actions without master filter,
select the workflow run you want to download artifact from, and download the binary from the Artifacts section.
The Docker image is available under kuskoman/logstash-exporter:<tag>.
You can pull the image using the following command:
docker pull kuskoman/logstash-exporter:<tag>
You can browse tags on the Docker Hub.
The Docker image can be run using the following command:
docker run -d \
-p 9198:9198 \
-e LOGSTASH_URL=http://logstash:9600 \
kuskoman/logstash-exporter:<tag>
The unstable version of the Docker image can be downloaded from the GitHub Container Registry. To pull the image from command line simply use:
docker pull ghcr.io/kuskoman/logstash-exporter:master
The Helm chart has its own README.
/metrics: Exposes metrics in Prometheus format./health: Returns 200 if app runs properly.
The application can be configured using the following environment variables, which are also loaded from .env file:
| Variable Name | Description | Default Value |
|---|---|---|
LOGSTASH_URL |
URL to Logstash API | http://localhost:9600 |
PORT |
Port on which the application will be exposed | 9198 |
HOST |
Host on which the application will be exposed | empty string |
LOG_LEVEL |
Log level | empty (defaults to info) |
All configuration variables can be checked in the config directory.
make all: Builds binary executables for all OS (Win, Darwin, Linux).make run: Runs the Go Exporter application.make build-linux: Builds a binary executable for Linux.make build-darwin: Builds a binary executable for Darwin.make build-windows: Builds a binary executable for Windows.make build-docker: Builds a Docker image for the Go Exporter application.make build-docker-multi: Builds a multi-arch Docker image (amd64andarm64).make clean: Deletes all binary executables in the out directory.make test: Runs all tests.make test-coverage: Displays test coverage report.make compose: Starts a Docker-compose configuration.make wait-for-compose: Starts a Docker-compose configuration until it's ready.make compose-down: Stops a Docker-compose configuration.make verify-metrics: Verifies the metrics from the Go Exporter application.make pull: Pulls the Docker image from the registry.make logs: Shows logs from the Docker-compose configuration.make minify: Minifies the binary executables.make install-helm-readme: Installs readme-generator-for-helm tool.make helm-readme: Generates Helm chart README.md file.make clean-elasticsearch: Cleans Elasticsearch data, works only with default ES port. The command may take a very long time to complete.make help: Shows info about available commands.
The main Go Exporter application is located in the cmd/exporter/main.go file. The binary executables are saved in the out directory.
Builds binary executables for all OS (Win, Darwin, Linux):
make all
Runs the Go Exporter application:
make run
Builds a binary executable for Linux:
make build-linux
Builds a binary executable for Darwin:
make build-darwin
Builds a binary executable for Windows:
make build-windows
Builds a Docker image for the Go Exporter application:
make build-docker
Builds a multi-arch Docker image (amd64 and arm64):
make build-docker-multi
Deletes all binary executables in the out directory:
make clean
Runs all tests:
make test
Displays test coverage report:
make test-coverage
Starts a Docker-compose configuration:
make compose
Starts a Docker-compose configuration until it's ready:
make wait-for-compose
Stops a Docker-compose configuration:
make compose-down
Verifies the metrics from the Go Exporter application:
make verify-metrics
Pulls the Docker image from the registry:
make pull
Shows logs from the Docker-compose configuration:
make logs
Minifies the binary executables:
make minify
Installs readme-generator-for-helm tool:
make install-helm-readme
Generates Helm chart README.md file:
make helm-readme
Cleans Elasticsearch data, works only with default ES port. The command may take a very long time to complete:
make clean-elasticsearch
Shows info about available commands:
make help
Application repository contains some helper scripts, which can be used to improve process of building, testing, and running the application. These scripts are not useful for the end user, but they can be useful for all potential contributors. The helper scripts are located in the scripts directory.
This script is used to add metrics table to the README.md file. Usage:
./scripts/add_metrics_to_readme.sh
This script is used to create release notes for the GitHub release. Used primarily by the CI workflow.
This script is used to generate Helm chart README.md file. The readme contains all the configuration variables from the values.yaml file.
This script is used to install readme-generator-for-helm tool. The tool is used to generate Helm chart README.md file. The script installs the tool under helm-generator directory.
This script is used to verify the metrics from the Go Exporter application. Can be used both locally and in the CI workflow.
./scripts/verify_metrics.sh
The application contains both unit and integration tests. All the tests are executed in the CI workflow.
Unit tests are located in the same directories as the tested files. To run all unit tests, use the following command:
make test
Integration tests checks if Prometheus metrics are exposed properly. To run them you must setup development docker-compose file.
make wait-for-compose
Then you can run the tests:
make verify-metrics
A Grafana Dashboard designed for metrics from Logstash-exporter on Kubernetes is available at grafana.com/grafana/dashboards/18628-logstash-on-kubernetes-dashboard/. This dashboard's JSON source is at excalq/grafana-logstash-kubernetes.
(If not using Kubernetes, change $pod to $instance in the JSON.)
This projects code was reviewed by Boldly Go in an awesome video, which in a huge way helped me to improve the code quality.
If you want to contribute to this project, please read the CONTRIBUTING.md file.
Table of exported metrics:
| Name | Type | Description |
|---|---|---|
| logstash_exporter_build_info | gauge | A metric with a constant '1' value labeled by version, revision, branch, goversion from which logstash_exporter was built, and the goos and goarch for the build. |
| logstash_info_build | counter | A metric with a constant '1' value labeled by build date, sha, and snapshot of the logstash instance. |
| logstash_info_node | counter | A metric with a constant '1' value labeled by node name, version, host, http_address, and id of the logstash instance. |
| logstash_info_pipeline_batch_delay | counter | Amount of time to wait for events to fill the batch before sending to the filter and output stages. |
| logstash_info_pipeline_batch_size | counter | Number of events to retrieve from the input queue before sending to the filter and output stages. |
| logstash_info_pipeline_workers | counter | Number of worker threads that will process pipeline events. |
| logstash_info_status | counter | A metric with a constant '1' value labeled by status. |
| logstash_info_up | gauge | A metric that returns 1 if the node is up, 0 otherwise. |
| logstash_stats_events_duration_millis | gauge | Duration of events processing in milliseconds. |
| logstash_stats_events_filtered | gauge | Number of events filtered out. |
| logstash_stats_events_in | gauge | Number of events received. |
| logstash_stats_events_out | gauge | Number of events out. |
| logstash_stats_events_queue_push_duration_millis | gauge | Duration of events push to queue in milliseconds. |
| logstash_stats_flow_filter_current | gauge | Current number of events in the filter queue. |
| logstash_stats_flow_filter_lifetime | gauge | Lifetime number of events in the filter queue. |
| logstash_stats_flow_input_current | gauge | Current number of events in the input queue. |
| logstash_stats_flow_input_lifetime | gauge | Lifetime number of events in the input queue. |
| logstash_stats_flow_output_current | gauge | Current number of events in the output queue. |
| logstash_stats_flow_output_lifetime | gauge | Lifetime number of events in the output queue. |
| logstash_stats_flow_queue_backpressure_current | gauge | Current number of events in the backpressure queue. |
| logstash_stats_flow_queue_backpressure_lifetime | gauge | Lifetime number of events in the backpressure queue. |
| logstash_stats_flow_worker_concurrency_current | gauge | Current number of workers. |
| logstash_stats_flow_worker_concurrency_lifetime | gauge | Lifetime number of workers. |
| logstash_stats_jvm_mem_heap_committed_bytes | gauge | Amount of heap memory in bytes that is committed for the Java virtual machine to use. |
| logstash_stats_jvm_mem_heap_max_bytes | gauge | Maximum amount of heap memory in bytes that can be used for memory management. |
| logstash_stats_jvm_mem_heap_used_bytes | gauge | Amount of used heap memory in bytes. |
| logstash_stats_jvm_mem_heap_used_percent | gauge | Percentage of the heap memory that is used. |
| logstash_stats_jvm_mem_non_heap_committed_bytes | gauge | Amount of non-heap memory in bytes that is committed for the Java virtual machine to use. |
| logstash_stats_jvm_mem_pool_committed_bytes | gauge | Amount of bytes that are committed for the Java virtual machine to use in a given JVM memory pool. |
| logstash_stats_jvm_mem_pool_max_bytes | gauge | Maximum amount of bytes that can be used in a given JVM memory pool. |
| logstash_stats_jvm_mem_pool_peak_max_bytes | gauge | Highest value of bytes that were used in a given JVM memory pool. |
| logstash_stats_jvm_mem_pool_peak_used_bytes | gauge | Peak used bytes of a given JVM memory pool. |
| logstash_stats_jvm_mem_pool_used_bytes | gauge | Currently used bytes of a given JVM memory pool. |
| logstash_stats_jvm_threads_count | gauge | Number of live threads including both daemon and non-daemon threads. |
| logstash_stats_jvm_threads_peak_count | gauge | Peak live thread count since the Java virtual machine started or peak was reset. |
| logstash_stats_jvm_uptime_millis | gauge | Uptime of the JVM in milliseconds. |
| logstash_stats_pipeline_dead_letter_queue_dropped_events | counter | Number of events dropped by the dead letter queue. |
| logstash_stats_pipeline_dead_letter_queue_expired_events | counter | Number of events expired in the dead letter queue. |
| logstash_stats_pipeline_dead_letter_queue_max_size_in_bytes | counter | Maximum size of the dead letter queue in bytes. |
| logstash_stats_pipeline_dead_letter_queue_size_in_bytes | counter | Current size of the dead letter queue in bytes. |
| logstash_stats_pipeline_events_duration | counter | Time needed to process event. |
| logstash_stats_pipeline_events_filtered | counter | Number of events that have been filtered out by this pipeline. |
| logstash_stats_pipeline_events_in | counter | Number of events that have been inputted into this pipeline. |
| logstash_stats_pipeline_events_out | counter | Number of events that have been processed by this pipeline. |
| logstash_stats_pipeline_events_queue_push_duration | counter | Time needed to push event to queue. |
| logstash_stats_pipeline_flow_filter_current | gauge | Current number of events in the filter queue. |
| logstash_stats_pipeline_flow_filter_lifetime | counter | Lifetime number of events in the filter queue. |
| logstash_stats_pipeline_flow_input_current | gauge | Current number of events in the input queue. |
| logstash_stats_pipeline_flow_input_lifetime | counter | Lifetime number of events in the input queue. |
| logstash_stats_pipeline_flow_output_current | gauge | Current number of events in the output queue. |
| logstash_stats_pipeline_flow_output_lifetime | counter | Lifetime number of events in the output queue. |
| logstash_stats_pipeline_flow_queue_backpressure_current | gauge | Current number of events in the backpressure queue. |
| logstash_stats_pipeline_flow_queue_backpressure_lifetime | counter | Lifetime number of events in the backpressure queue. |
| logstash_stats_pipeline_flow_worker_concurrency_current | gauge | Current number of workers. |
| logstash_stats_pipeline_flow_worker_concurrency_lifetime | counter | Lifetime number of workers. |
| logstash_stats_pipeline_plugin_bulk_requests_errors | counter | Number of bulk request errors. |
| logstash_stats_pipeline_plugin_bulk_requests_responses | counter | Bulk request HTTP response counts by code. |
| logstash_stats_pipeline_plugin_documents_non_retryable_failures | counter | Number of output events with non-retryable failures. |
| logstash_stats_pipeline_plugin_documents_successes | counter | Number of successful bulk requests. |
| logstash_stats_pipeline_plugin_events_duration | counter | Time spent processing events in this plugin. |
| logstash_stats_pipeline_plugin_events_in | counter | Number of events received this pipeline. |
| logstash_stats_pipeline_plugin_events_out | counter | Number of events output by this pipeline. |
| logstash_stats_pipeline_plugin_events_queue_push_duration | counter | Time spent pushing events into the input queue. |
| logstash_stats_pipeline_queue_events_count | counter | Number of events in the queue. |
| logstash_stats_pipeline_queue_events_queue_size | counter | Number of events that the queue can accommodate |
| logstash_stats_pipeline_queue_max_size_in_bytes | counter | Maximum size of given queue in bytes. |
| logstash_stats_pipeline_reloads_failures | counter | Number of failed pipeline reloads. |
| logstash_stats_pipeline_reloads_successes | counter | Number of successful pipeline reloads. |
| logstash_stats_pipeline_reloads_last_failure_timestamp | gauge | Timestamp of last failed pipeline reload. |
| logstash_stats_pipeline_reloads_last_success_timestamp | gauge | Timestamp of last successful pipeline reload. |
| logstash_stats_pipeline_up | gauge | Whether the pipeline is up or not. |
| logstash_stats_process_cpu_load_average_1m | gauge | Total 1m system load average. |
| logstash_stats_process_cpu_load_average_5m | gauge | Total 5m system load average. |
| logstash_stats_process_cpu_load_average_15m | gauge | Total 15m system load average. |
| logstash_stats_process_cpu_percent | gauge | CPU usage of the process. |
| logstash_stats_process_cpu_total_millis | gauge | Total CPU time used by the process. |
| logstash_stats_process_max_file_descriptors | gauge | Limit of open file descriptors. |
| logstash_stats_process_mem_total_virtual | gauge | Total virtual memory used by the process. |
| logstash_stats_process_open_file_descriptors | gauge | Number of currently open file descriptors. |
| logstash_stats_queue_events_count | gauge | Number of events in the queue. |
| logstash_stats_reload_failures | gauge | Number of failed reloads. |
| logstash_stats_reload_successes | gauge | Number of successful reloads. |