Prometheus Aggregation Gateway

Prometheus Aggregation Gateway is a push gateway that aggregates metrics for Prometheus. As opposed to the official Prometheus Pushgateway, this service aggregates the sample values it receives.

Counters where all labels match are added up.
Histograms are added up; if bucket boundaries are mismatched then the result has the union of all buckets and counts are given to the lowest bucket that fits.
Gauges are also added up (but this may not make any sense)
Summaries are treated as a pair of counters (quantile information is discarded if present).

How to use

Send metrics in Prometheus format to /metrics/

E.g. if you have the program running locally:

echo 'http_requests_total{method="post",code="200"} 1027' | curl --data-binary @- http://localhost/metrics/

You can also add labels that will apply to all metrics that are passed. For example, the following will add the domain="sometest.com" and instance="nginx-1" labels to both metrics:

echo '
http_requests_total{method="post",code="200"} 1027
http_errors_total{method="post",code="500"} 6
' | curl --data-binary @- http://localhost/metrics/domain/sometest.com/instance/nginx-1

Now you can push your metrics using your favorite Prometheus client.

E.g. in Python using prometheus/client_python:

from prometheus_client import CollectorRegistry, Counter, push_to_gateway
registry = CollectorRegistry()
counter = Counter('some_counter', "A counter", registry=registry)
counter.inc()
push_to_gateway('localhost', job='my_job_name', registry=registry)

Then have your Prometheus scrape metrics at /metrics.

Running the service

help command can be useful when calling the binary to see what commands and flags are available

$ prom-aggregation-gateway --help

prometheus aggregation gateway

Usage:
  prom-aggregation-gateway [flags]
  prom-aggregation-gateway [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  start       starts up the server
  version     Show version information

Flags:
      --AuthUsers strings        List of allowed auth users and their passwords comma separated
                                  Example: "user1=pass1,user2=pass2"
      --apiListen string         Listen for API requests on this host/port. (default ":80")
      --cors string              The 'Access-Control-Allow-Origin' value to be returned. (default "*")
  -h, --help                     help for prom-aggregation-gateway
      --lifecycleListen string   Listen for lifecycle requests (health, metrics) on this host/port (default ":8888")

Use "prom-aggregation-gateway [command] --help" for more information about a command.

Any flags you see above can also be set by ENV_VARIABLES. ENV_VARS must have a prefix of PAG_, for example PAG_AUTHUSERS=user1=pass1,user2=pass2 will start the service with basic auth. If an ENV_VARIABLE is set than it will be used over a CLI argument passed to the service.

Ready-built images

Container images are published here:

https://ghcr.io/zapier/prom-aggregation-gateway

Helm Chart

Helm Charts are published here:

https://zapier.github.io/prom-aggregation-gateway/

You can use them:

helm repo add pag https://zapier.github.io/prom-aggregation-gateway/
helm repo update
helm search repo pag -l

Helm Chart Unit Tests

Helm plugin quintush/helm-unittest is used for unit-testing.
After installing the plugin run helm unittest <chart_dir>. For Helm 3 support use --helm3 flag.
More information on writing tests might be found in the plugin's project repo.

Contributing

To run the server you can run:

go run .

Testing

To run the tests you can run:

go test

VSCode

To debug locally you can setup a launch.json with the following:

{
    "version": "0.2.0",
    "configurations": [
      {
        "name": "Debug prom-agg-gateway",
        "type": "go",
        "request": "launch",
        "mode": "debug",
        "program": "${workspaceFolder}",
        "args": [
          "run",
          "."
        ]
      }
    ]
  }

Then you'll be able to launch prom-agg-gateway locally and debug it from within VSCode. You must have Delve installed locally for this to work.

If you want a debugger that will launch the tests you can create this configuration:

{
    "version": "0.2.0",
    "configurations": [
      {
        "name": "Test Current File",
        "type": "go",
        "request": "launch",
        "mode": "test",
        "program": "${workspaceFolder}/${relativeFileDirname}",
        "showLog": true
      }
    ]
  }

Comparison to Prometheus Pushgateway

According to https://prometheus.io/docs/practices/pushing/:

The Pushgateway never forgets series pushed to it and will expose them to Prometheus forever...

The latter point is especially relevant when multiple instances of a job differentiate their metrics in the Pushgateway via an instance label or similar.

This restriction makes the Prometheus pushgateway inappropriate for the use case of accepting metrics from a client-side web app, so we created this one to aggregate counters from multiple senders.

Prom-aggregation-gateway presents a similar API but does not attempt to be a drop-in replacement.

Client Libraries

Python

https://github.com/prometheus/client_python

JS

Getting Help

If you have any questions about, feedback for or problems with prom-aggregation-gateway:

File an issue.

prom-aggregation-gateway follows the CNCF Code of Conduct. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting a project maintainer.

Your feedback is always welcome!

orthly/prom-aggregation-gateway