alexvaut/cLoki

Clickhouse Loki: Grafana Loki API + ClickHouse Backend in NodeJS

cLoki

like Loki, but for Clickhouse.

Super experimental, fully functional Loki API emulator made with NodeJS, Fastify and Clickhouse
APIs are 100% Compatible with Grafana Explore and paStash for logs ingestion

🔥 Beta Stage, Contributions Welcome! Do not use this for anything serious.. yet!

Project Background

The Loki API is brilliantly simple and appealing - its misteriously assembled Cortex backend, not so much. We wanted to leverage this concept over a true and fully open-source solution end-to-end. Also, we love Clickhouse .

cLoki implements the same API functionality as Loki, buffered by a fast bulking LRU sitting on top of Clickhouse and relying on its columnar search and insert performance alongside solid distribuion and clustering capabilities for stored data. Just like Loki, cLoki does not parse or index incoming logs, but rather groups log streams using the same label system as Prometheus.

The current purpose of this project is to research and understand inner aspects of the original implementation.

---

Setup

🚏 Manual

Clone this repository, install with npmand run using nodejs 8.x (or higher)

npm install
npm start

🚏 Docker

For a fully working demo, check the docker-compose example

Configuration

The following ENV Variables can be used to control cLoki parameters and backend settings.

ENV	Default	Usage
CLICKHOUSE_SERVER	localhost	Clickhouse Server address
CLICKHOUSE_PORT	8123	Clickhouse Server port
CLICKHOUSE_DB	default	Clickhouse Database Name
CLICKHOUSE_TSDB	loki	Clickhouse TS Database Name
CLICKHOUSE_AUTH	default:	Clickhouse Authentication (user:password)
BULK_MAXAGE	2000	Max Age for Bulk Inserts
BULK_MAXSIZE	5000	Max Size for Bulk Inserts
BULK_MAXCACHE	50000	Max Labels in Memory Cache
HOST	0.0.0.0	cLOKi API IP
PORT	3100	cLOKi API PORT
CLOKI_LOGIN	false	Basic HTTP Username
CLOKI_PASSWORD	false	Basic HTTP Password
DEBUG	false	Debug Mode

⛽ Log Streams

The ideal companion for parsing and shipping log streams to cLoki is paStash with extensive interpolation capabilities.

---

Project Status

API

Loki API Functions are loosely implemented as documented by the Loki API reference.

• /api/prom/push
• /api/prom/query
• /api/prom/label
• /api/prom/label/name/values

Status

• Basic Writes
  • Label Fingerprints
  • Sample Series
  • JSON Support
  • ProtoBuf Support
• Basic Fingerprinting
• Stream Selector rules ()
  • = exactly equal.
  • != not equal.
  • =~ regex-match.
  • !~ do not regex-match.
• Basic Search
  • Labels (single key, multi key, AND logic)
  • Samples (by Fingerprint match)

---

API Examples

INSERT Labels & Logs

# curl --header "Content-Type: application/json" --request POST \
  --data '{"streams":[{"labels":"{\"__name__\":\"up\"}","entries":[{"timestamp":"2018-12-26T16:00:06.944Z","line":"zzz"}]} \ http://localhost:3100/api/prom/push

QUERY Logs

# curl 'localhost:3100/api/prom/query?query={__name__="up"}'

{"streams":[{"labels":"{\"__name__\":\"up\"}","entries":[{"timestamp":"1545840006944","line":"zzz"},{"timestamp":"1545840006944","line":"zzz"},{"timestamp":"1545840006944","line":"zzz"}]}]}root@de4 ~ #

QUERY Labels

# curl 'localhost:3100/api/prom/label'

{"values":["__name__"]}

QUERY Label Values

# curl 'localhost:3100/api/prom/label/__name__/values'

{"values":["up"]}

---

Database Schema

CREATE TABLE time_series (
    date Date,
    fingerprint UInt64,
    labels String,
    name String
)
ENGINE = ReplacingMergeTree
    PARTITION BY date
    ORDER BY fingerprint;

CREATE TABLE samples (
    fingerprint UInt64,
    timestamp_ms Int64,
    value Float64,
    string String,
)
ENGINE = MergeTree
    PARTITION BY toDate(timestamp_ms / 1000)
    ORDER BY (fingerprint, timestamp_ms);

Raw Queries

CREATE

DATABASE

CREATE DATABASE IF NOT EXISTS loki

TABLES

CREATE TABLE IF NOT EXISTS loki.time_series (date Date,fingerprint UInt64,labels String, name String) ENGINE = ReplacingMergeTree PARTITION BY date ORDER BY fingerprint;
CREATE TABLE IF NOT EXISTS loki.samples (fingerprint UInt64,timestamp_ms Int64,value Float64,string String) ENGINE = MergeTree PARTITION BY toRelativeHourNum(toDateTime(timestamp_ms / 1000)) ORDER BY (fingerprint, timestamp_ms);

SELECT

FINGERPRINTS

SELECT DISTINCT fingerprint, labels FROM loki.time_series

SAMPLES

SELECT fingerprint, timestamp_ms, string
	FROM loki.samples
	WHERE fingerprint IN (7975981685167825999) AND timestamp_ms >= 1514730532900 
	AND timestamp_ms <= 1514730532902
	ORDER BY fingerprint, timestamp_ms

SELECT fingerprint, timestamp_ms, value
	FROM loki.samples
	ANY INNER JOIN 7975981685167825999 USING fingerprint
	WHERE timestamp_ms >= 1514730532900 AND timestamp_ms <= 1514730532902
	ORDER BY fingerprint, timestamp_ms

INSERT

FINGERPRINTS

INSERT INTO loki.time_series (date, fingerprint, labels, name) VALUES (?, ?, ?, ?) 

SAMPLES

INSERT INTO loki.samples (fingerprint, timestamp_ms, value, string) VALUES (?, ?, ?, ?)

---

Acknowledgements

cLoki is not affiliated or endorsed by Grafana Labs. All rights belong to their respective owners.