An example REST API with prometheus metrics integration. The REST API is built using flask.
The prometheus metrics integration is done using prometheus-flask-exporter built on prometheus-client for python. This example application tries to cover the basic metrics collections features using these libraries. Refer the respective documentation for advanced use cases.
Refer the About the Integration section to learn how the integration is done in the code.
Similar libaries are available for other languages and frameworks:
- express-prometheus-middleware
- django-promethues
- Promethues Client libraries: https://prometheus.io/docs/instrumenting/clientlibs/
If you're new to Prometheus refer the confluence guide: Getting Started with Prometheus.
The entire applicatio can be run and tested locally. Prometheus should be installed already in the machine. Follow confluence guide for installation steps.
- Create virtual environment:
python -m venv venv - Activate environment:
source .\venv\Scripts\activate - Install dependencies:
pip install -r env/pip/requirements.txt
-
Copy the
prometheus.ymlconfiguraton to prometheus installation path. Windows:cp prometheus.yml ~/bin/prometheus-2.32.1.windows-amd64/Linux:
cp prometheus.yml ~/bin/prometheus-2.32.1.linux-amd64/ -
Start prometheus server. Refer the confluence guide for running prometheus.
-
Run the application.
python -u run.py
The app also included configurations for gunicorn. To run with gunicorn execute the command.
./run.sh
Hit the APIs:
curl -s http://localhost:5000/ --header "user: john"
curl -s http://localhost:5000/square?num=20 --header "user: john"
curl -s http://localhost:5000/even?num=20 --header "user: john"The app is currently deployed in SBD Caspian cluster. Endpoint: https://sbd-prometheus-flask-app.sandbox.sbd-caspian.com/
Open the /metrics endpoint to see the prometheus metrics in plain text format: http://localhost:5000/metrics
If running with gunicorn open: http://localhost:9200/metrics
Open prometheus UI and query the metrics.
Example metrics:
flask_http_request_duration_seconds_count
flask_app_even_numbers_bucket
sum(flask_http_request_duration_seconds_count{app="sbd-prometheus-flask-app"})
Jupyterhub notebook with examples on querying prometheus with Python: notebook
The Prometheus integration is primarily done through prometheus-flask-exporter. It exposes HTTP metrics by default for all flask routes. It has the necessary helper classes to integrate with Gunicorn.
An object of prometheus-flask-exporter class PrometheusMetrics is initialized in metrics.py. It creates a Flask app factory object which is then imported and registered in the Flask app in app.py. This is the only step required for exposing the default HTTP metrics for the Flask app.
When run with Gunicorn the GunicornPrometheusMetrics is used instead of PrometheusMetrics. Gunicorn should be run with config.py.
We can add custom metrics for all flask routes. The register_metrics function in the app.py adds a custom metric flask_request_in_progress for all flask routes. It's uses to track the number of active requests for each request path.
Custom metrics can be added for specific flask routes. Refer the github example
Default metrics can be excluded for certain routes by using do_not_track decorator as seen for the function health in routes.py.
For creating metrics which aren't related to Flask routes the prometheus-client for python can be used. total_auths is a counter metrics added in routes.py in the authenticate function using the prometheus-client library. Remember to use the registry imported by metrics.py when new metrics are created. This ensures the metrics are added along with ones from prometheus-flask-exporter.
For more examples on usage of the library refer prometheus-flask-exporter and prometheus-client.
Debug mode on Flask breask the prometheus-flask-exporter module. Turn off debug mode to see the metrics.
if __name__ == "__main__":
app.run(debug=False)