/Prometheus-AzureDataExplorer

Prometheus remote storage adapter for Azure Data Explorer

Primary LanguageC#MIT LicenseMIT

Prometheus adapter for Azure Data Explorer

OSS-TODO Build and test Deploy

Glossary

Overview

This is a production-ready implementation of Prometheus remote storage adapter for Azure Data Explorer (a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming).

The idea and first start for this project (previously it was a fork) became possible thanks to PrometheusToAdx repository.

The major functionality features:

  1. Uses azure funtions;
  2. Fully workable remote read/write;
  3. Possible to write a large amounts of metrics because of ingesting data via sets of blobstorage -> eventgrid -> eventhub chain (see integration principe bellow);
  4. Does downsampling for old metrics for save db size.

Information links:

Integration principe

Ingestion flow miro

How to run

1. Required resources table

Resource Resource Type
<cluster>-prometheus-adx Service Principal
<cluster>-prometheus-adx Function App
<cluster>-adx Azure Data Explorer Cluster
<cluster>prometheusadx Storage Account
<cluster>-prometheus-adx Event Hub Namespace
<cluster>-prometheus-adx Event Hub Instance
<cluster>-prometheus-adx Event Grid Subscription

2. Step-by-step guide

  1. Create Service Principal account;
  2. Create Storage Account;
  3. Create Azure Data Explorer Cluster;
  4. Create Function App;
  5. Add Contributor for Service Principal account;
  6. Add Admin for Azure Data Explorer Cluster database (only for database);
  7. Create Event Hub Namespace with Event Hub Instance
  8. Create Event Grid Subscription with following options:
    • Topic type - Storage Accounts
    • Topic resource - Storage Account (2)
    • Filter event type - Blob created
    • Endpoint details - Event Hub (7)
  9. Create database <cluster>-prometheus in Azure Data Explorer Cluster:
    • Permissions: Add View role for Service Principal (1)
    • Create tables RawData, Metrics (KQL bellow)
    • Data injestion:
      • Connection type - Blob storage
      • Storage Account / Event Grid - Storage Account (2) and Event Grid Subscription (8)
      • Event type - Blob created
      • Table - Metrics
      • Data format - CSV
      • Mapping name - CsvMapping

Init tables KQLs

.drop table Metrics ifexists
.drop table RawData ifexists

.create table RawData (Datetime: datetime, Timestamp: long, Name: string, Instance: string, Job: string, Labels: dynamic, LabelsHash: long, Value: real)

.create-or-alter table RawData ingestion csv mapping 'CsvMapping'
'['
'   { "column" : "Datetime", "DataType":"datetime", "Properties":{"Ordinal":"0"}},'
'   { "column" : "Timestamp", "DataType":"long", "Properties":{"Ordinal":"1"}},'
'   { "column" : "Name", "DataType":"string", "Properties":{"Ordinal":"2"}},'
'   { "column" : "Instance", "DataType":"string", "Properties":{"Ordinal":"3"}},'
'   { "column" : "Job", "DataType":"string", "Properties":{"Ordinal":"4"}},'
'   { "column" : "Labels", "DataType":"dynamic", "Properties":{"Ordinal":"5"}},'
'   { "column" : "LabelsHash", "DataType":"long", "Properties":{"Ordinal":"6"}},'
'   { "column" : "Value", "DataType":"real", "Properties":{"Ordinal":"7"}},'
']'

.alter-merge table RawData policy retention softdelete = 4d recoverability = disabled

.create table Metrics (LabelsHash: long, StartDatetime: datetime, EndDatetime: datetime, Name: string, Instance: string, Job: string, Labels: dynamic, Samples: dynamic)

How to develop

This repo contains .devcontainer for VSCode, which runs a set of containers with:

  • Prometheus server as is
  • Prometheus server with remote_read/remote_write for local debug
  • Grafana
  • Node-exporter node for some example data
  • Dev-container with .net core for azure functions installation with all requirements and vscode extensions for C# development

Before run within .devcontainer:

You will need to setup some credentials to fully debug the functions. There are 2 ways for doing this:

  • Check the .devcontainer/devcontainer.json file > remote Env section for environment vars.
  • Or you can change the ./src/PromADX/local.settings.json config file. Don't forget to exclude this file from commit.

Prometheus protocol buffer generator

  • This repo uses prometheus .proto and convert them via protoc util to C# classes
  • Follow this documentation and scripts to get, compile and keep up the latest versions

Additional information

Prometheus read/write format

Samples:[
	{
		timestamp:
		value:
	},
	{
		timestamp:
		value:
	}...
],
Labels:[
    {
		name:
		value:
	},
	{
		name:
		value:
	}...
]

Get metrics KQL example

Metrics
| where (EndDatetime >= unixtime_milliseconds_todatetime(1591084670098)) and (StartDatetime <= unixtime_milliseconds_todatetime(1591092170098)) and ( ( Name == 'mysql_global_status_queries' ) )
| summarize Labels=tostring(any(Labels)), Samples=make_list( Samples ) by LabelsHash
| mv-apply Samples = Samples on
(
    order by tolong(Samples['Timestamp']) asc
    | summarize Samples=make_list(pack('Timestamp', Samples['Timestamp'], 'Value', Samples['Value']))
)