timescaleproxy
A proxy for writing Telegraf JSON HTTP outputs into a TimescaleDB. Written in Go. Feedback is welcome. Bear in mind this is my first Go-project.
As the Telegraf plugin for TimescaleDB is pending pending for being included in the Telegraf codebase.
I saw the need for a workaround using a HTTP-proxy that recieves metrics in JSON and writes them to the TimescaleDB.
It's inspired by the plugin but instead of a general PostgreSQL output it outputs only to TimescaleDB.
Security is non-existant at the moment so this should only be used far away from the internetz. I'd recommend either behind a secured nginx reverse proxy or over a wireguard interface.
Install
git clone https://github.com/dafvid/timescaleproxy.git
cd timescaleproxy
go install
go build
Config
./timescaleproxy -printconf > config.json
Db
- Db.Host: PostgreSQL server host
- Db.Port: PostgreSQL port
- Db.Schema: Timescale database schema
- Db.Database: Timescale database name
- Db.User: Timescale database user
- Db.Password: Timescale database user password
- Db.MaxConns: MaxConns setting for pgxpool
- Db.MinConns: MinConns setting for pgxpool
Listen
- Listen.adress: Timescale proxy listening adress
- Listen.port: Timescale proxy listening port
Misc
- TimestampUnit: matches the setting
json_timestamp_units
in section[[outputs.http]]
intelegraf.conf
- DefaultDropPolicy: Interval setting for add_retention_policy(). Clear to disable retention.
- LogLevel: Log level. Clear to disable logs.
Run
./timescaleproxy -c config.json
Telegraf config
[[outputs.http]]
url = "http://url.to.server:8432/"
data_format = "json"
json_timestamp_units = "1ms"
[outputs.http.headers]
Content-Type = "application/json; charset=utf-8"
Updates
2022-10-07
- Added Config documentation
2022-08-10
- Added FreeBSD RC-script to run the proxy as a daemon
2022-06-22
- Changed the timestamp column name to ts
2020-07-31:
- Export and read JSON-config
- Create tables and tags in DB from first Metric
- Writes JSON-metrics to TimescaleDB
TODO
- Per measurement config for retention
configcreate tables from first measurementwrite measurement to dbtags as FKschema config- handle influx line protocol (less portable maybe)
- measurement config (column type for field)
- sanitize strings
default retention policyerror handling- update table if measurement changes (unlikely)