Prometheus exporter for Azure LogAnalytics Kusto queries with configurable fields and transformations.
azure-loganalytics-exporter can query configured workspaces or all workspaces in one or multiple subscriptions.
The exporter can also cache metrics and servicediscovery information to reduce requests against workspaces and Azure API.
Usage:
azure-loganalytics-exporter [OPTIONS]
Application Options:
--log.debug debug mode [$LOG_DEBUG]
--log.devel development mode [$LOG_DEVEL]
--log.json Switch log output to json format [$LOG_JSON]
--azure.environment= Azure environment name (default: AZUREPUBLICCLOUD) [$AZURE_ENVIRONMENT]
--azure.servicediscovery.cache= Duration for caching Azure ServiceDiscovery of workspaces to reduce API calls (time.Duration)
(default: 30m) [$AZURE_SERVICEDISCOVERY_CACHE]
--azure.resource-tag= Azure Resource tags (space delimiter) (default: owner) [$AZURE_RESOURCE_TAG]
--loganalytics.workspace= Loganalytics workspace IDs [$LOGANALYTICS_WORKSPACE]
--loganalytics.concurrency= Specifies how many workspaces should be queried concurrently (default: 5) [$LOGANALYTICS_CONCURRENCY]
-c, --config= Config path [$CONFIG]
--server.bind= Server address (default: :8080) [$SERVER_BIND]
--server.timeout.read= Server read timeout (default: 5s) [$SERVER_TIMEOUT_READ]
--server.timeout.write= Server write timeout (default: 10s) [$SERVER_TIMEOUT_WRITE]
Help Options:
-h, --help Show this help message
for Azure API authentication (using ENV vars) see https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication
- see example.yaml
| Endpoint | Description |
|---|---|
/query |
Query tester |
/metrics |
Default prometheus golang metrics |
/probe |
Execute loganalytics queries against workspaces (set on commandline/env var) |
/probe/workspace |
Execute loganalytics queries against workspaces (defined as parameter) |
/probe/subscription |
Execute loganalytics queries against workspaces (using servicediscovery) |
HINT: parameters of type multiple can be either specified multiple times and/or splits multiple values by comma.
uses predefined workspace list defined as parameter/environment variable on startup
| GET parameter | Default | Required | Multiple | Description |
|---|---|---|---|---|
module |
no | no | Filter queries by module name | |
cache |
no | no | Use of internal metrics caching (time.Duration) | |
parallel |
$LOGANALYTICS_PARALLEL |
no | no | Number (int) of how many workspaces can be queried at the same time |
uses dynamically passed workspaces via HTTP query parameter
| GET parameter | Default | Required | Multiple | Description |
|---|---|---|---|---|
module |
no | no | Filter queries by module name | |
workspace |
yes | yes | Workspace IDs which are probed | |
cache |
no | no | Use of internal metrics caching (time.Duration) | |
parallel |
$LOGANALYTICS_PARALLEL |
no | no | Number (int) of how many workspaces can be queried at the same time |
uses Azure service discovery to find all workspaces in one or multiple subscriptions
| GET parameter | Default | Required | Multiple | Description |
|---|---|---|---|---|
module |
no | no | Filter queries by module name | |
subscription |
yes | yes | Uses all workspaces inside subscription | |
filter |
no | no | Advanced filter for resource | {filter} | project id, customerId=properties.customerId ResoruceGraph query (available with 23.6.0) |
|
cache |
no | no | Use of internal metrics caching (time.Duration) | |
parallel |
$LOGANALYTICS_PARALLEL |
no | no | Number (int) of how many workspaces can be queried at the same time |
optional |
false |
no | no | Do not fail, if service discovery did not find any workspaces |
available on /metrics
| Metric | Description |
|---|---|
azure_loganalytics_status |
Status if query was successfull (per workspace, module, metric) |
azure_loganalytics_last_query_successfull |
Timestamp of last successfull query (per workspace, module, metric) |
azure_loganalytics_query_time |
Summary metric about query execution time (incl. all subqueries) |
azure_loganalytics_query_results |
Number of results from query |
azure_loganalytics_query_requests |
Count of requests (eg paged subqueries) per query |
azure_loganalytics_workspace_query_count |
Count of discovered workspaces per module |
(with 22.2.0 and later)
Azuretracing metrics collects latency and latency from azure-sdk-for-go and creates metrics and is controllable using environment variables (eg. setting buckets, disabling metrics or disable autoreset).
| Metric | Description |
|---|---|
azurerm_api_ratelimit |
Azure ratelimit metrics (only on /metrics, resets after query due to limited validity) |
azurerm_api_request_* |
Azure request count and latency as histogram |
| Environment variable | Example | Description |
|---|---|---|
METRIC_AZURERM_API_REQUEST_BUCKETS |
1, 2.5, 5, 10, 30, 60, 90, 120 |
Sets buckets for azurerm_api_request histogram metric |
METRIC_AZURERM_API_REQUEST_ENABLE |
false |
Enables/disables azurerm_api_request_* metric |
METRIC_AZURERM_API_REQUEST_LABELS |
apiEndpoint, method, statusCode |
Controls labels of azurerm_api_request_* metric |
METRIC_AZURERM_API_RATELIMIT_ENABLE |
false |
Enables/disables azurerm_api_ratelimit metric |
METRIC_AZURERM_API_RATELIMIT_AUTORESET |
false |
Enables/disables azurerm_api_ratelimit autoreset after fetch |
azurerm_api_request label |
Status | Description |
|---|---|---|
apiEndpoint |
enabled by default | hostname of endpoint (max 3 parts) |
routingRegion |
enabled by default | detected region for API call, either routing region from Azure Management API or Azure resource location |
subscriptionID |
enabled by default | detected subscriptionID |
tenantID |
enabled by default | detected tenantID (extracted from jwt auth token) |
resourceProvider |
enabled by default | detected Azure Management API provider |
method |
enabled by default | HTTP method |
statusCode |
enabled by default | HTTP status code |
see example.yaml for general ingestion metrics (number of rows per second and number of bytes per second per table)
see example.aks-single.yaml for AKS namespace ingestion metrics (number of rows per second and number of bytes per AKS namespace) sending queryies to each LogAnalaytics workspace individually (single mode)
see example.aks-multi.yaml for AKS namespace ingestion metrics (number of rows per second and number of bytes per AKS namespace) sending one query against multiple LogAnalaytics workspaces (multi mode)
more examples of result processing can be found within azure-resourcegraph-expoter (uses same processing library)
Config file:
queries:
- metric: azure_loganalytics_operationstatus_count
query: |-
Operation
| summarize count() by OperationStatus
fields:
- name: count_
type: value
Metrics:
# HELP azure_loganalytics_operationstatus_count azure_loganalytics_operationstatus_count
# TYPE azure_loganalytics_operationstatus_count gauge
azure_loganalytics_operationstatus_count{OperationStatus="Succeeded",workspaceId="xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx",workspaceTable="PrimaryResult"} 1
predefined workspaces (at startup via parameter/environment variable)
- job_name: azure-loganalytics-exporter
scrape_interval: 1m
metrics_path: /probe
params:
cache: ["10m"]
parallel: ["5"]
static_configs:
- targets: ["azure-loganalytics-exporter:8080"]dynamic workspaces (defined in prometheus configuration)
- job_name: azure-loganalytics-exporter
scrape_interval: 1m
metrics_path: /probe/workspace
params:
workspace:
- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
cache: ["10m"]
parallel: ["5"]
static_configs:
- targets: ["azure-loganalytics-exporter:8080"]find workspaces with servicediscovery via subscription
- job_name: azure-loganalytics-exporter
scrape_interval: 1m
metrics_path: /probe/subscription
params:
subscription:
- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx
cache: ["10m"]
parallel: ["5"]
static_configs:
- targets: ["azure-loganalytics-exporter:8080"]