Extend the prometheus exporter to include player metrics

[camalot]

Would like to see the prometheus exporter be extended to provide the info that is in /data/world/stats.

This is a pretty big enhancement, IMHO. But wanted to track it, discuss it, and explain how I am doing it today

My current custom implementation

I am going to explain what I have, just to give details. It is not the expected solution, as it requires many different tools that are not part of the minecraft system.

A custom solution that is a combination of bash on the host that runs a cronjob, a mongodb table to store the data, and a couple flows in nodered that put the data in the mongodb collection, and that serves up the metrics endpoint for prometheus.

This is a cronjob that runs every 5 minutes. It takes the contents of the stats directory and POSTs to a nodered endpoint that

#!/usr/bin/env bash

data_dir="/mnt/container_data/minecraft/curseforge/data";
stats_dir="${data_dir}/world/stats";

# loop all json files in $stats_dir
# the file name is the player uuid

for file in ${stats_dir}/*.json; do
  # get the player uuid
  echo "Processing ${file}";
  player_uuid=$(basename ${file} .json);
  
  stat_data=$(cat ${file});
  echo "Sending data to Node-RED";
  curl -X POST -H "Content-Type: application/json" -d "${stat_data}" "https://nodered.bit13.local/minecraft/player/${player_uuid}/stats/";
done

This posts the stats from each file, finds the user, from my whitelist, and then stores the full set of data in mongo.

var user_info = msg.payload[0];
var data = global.get("mc_stats")["stats"];
var timestamp = Date.now();
msg.payload = {
    "$set": {
        "uuid": user_info.uuid, // comes from whitelist lookup
        "user_id": user_info.user_id, // comes from whitelist lookup
        "username": user_info.username, // comes from whitelist lookup
        "stats": data,
        "modified": timestamp
    }
}

Besides the POST endpoint to collect the metrics, and store them in mongodb, there is a metrics endpoint that is consumed by prometheus.

This is what this flow looks like

The first line of that flow is the stats collection, and then the formatting that data to a prometheus exported data format.
The second line gets the installed mod version (ie: ATM8 1.0.15)
The third line gets the server stats, like max players, online players, etc.

The collect metrics step from the first line of the flow:

function safe_name(n) {
    return n.replace(/\:/gmi, "_");
}
function metric_name(group,metric,labels) {
    labels["metric"] = metric
    split_item = metric.split(":");
    if (split_item && split_item.length == 2) {
        labels["source"] = split_item[0];
        labels["item"] = split_item[1];
    }
    return `mcstats_${safe_name(group)}{${labels_out(labels)}}`.toLowerCase();
}
function labels_out(labels) {
    let outValue = "";
    for (let l in labels) {
        outValue += `${l}="${labels[l]}",`;
    }
    if (outValue.length >= 1) {
        outValue = outValue.slice(0, -1);
    }
    return outValue;
}
let metrics = {};

for (let x = 0; x < msg.payload.length; ++x) {
    var labels = {};
    var stats = msg.payload[x].stats;
    
    labels["uuid"] = msg.payload[x].uuid;
    labels["username"] = msg.payload[x].username;
    labels['user_id'] = msg.payload[x].user_id;
    for (let group in stats) {
        for (let metric in stats[group]) {
            metrics[metric_name(group,metric,labels)] = stats[group][metric];
        }
    }   
}
global.set("mc_stats_metrics", metrics);
msg.payload = metrics;
return msg;

What I have setup ties in to my whitelist, so I am able to get the username. I look up the uuid to get the username. I wouldn't expect it to be this level of info, as it requires a way to lookup the uuid -> username. (hitting an api for this would have to include caching). But having the username makes it very easy to filter by user on a grafana dashboard.

user_id: discord user id.
username: minecraft username
uuid: minecraft uuid
metric: the full metric name, defined in the stats
source: derived from the metric. what provides the metric
item: derived from the metric. what the metric is.

the prometheus metric name itself comes from the stats file. looping over the objects in the stats to create them. replacing : with _. Here is a screenshot of the metrics for a user.

The children of this become the prometheus exported metric.

mcstats_minecraft_killed{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="creeperoverhaul:hills_creeper",source="creeperoverhaul",item="hills_creeper"} 1
mcstats_minecraft_dropped{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:red_mushroom",source="minecraft",item="red_mushroom"} 2
mcstats_minecraft_used{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="create:brass_casing",source="create",item="brass_casing"} 22
mcstats_minecraft_custom{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="tombstone:kill_undead_boss",source="tombstone",item="kill_undead_boss"} 2
mcstats_minecraft_killed_by{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:ravager",source="minecraft",item="ravager"} 1
mcstats_minecraft_mined{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:cobbled_deepslate",source="minecraft",item="cobbled_deepslate"} 247
mcstats_minecraft_broken{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:stone_shovel",source="minecraft",item="stone_shovel"} 2
mcstats_minecraft_picked_up{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:enchanted_book",source="minecraft",item="enchanted_book"} 2
mcstats_minecraft_crafted{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:stone_brick_stairs",source="minecraft",item="stone_brick_stairs"} 28

Here are some screenshots of the Grafana Dashboard. Almost all of the panel items are filterable in some way. By user, by source, or by item. dashboard json

[itzg]

That is great feature idea, but will need to be a whole separate tool/application like what you have implemented. mc-monitor is intended only for remote/network based status retrieval. In theory, it can be used to monitor minecraft servers all over the internet.

[camalot]

That makes sense. I noticed that mc-monitor had the prometheus exporter, so I thought this would be a good place for it.

[itzg]

Yep, the idea has a lot similar facets. Good to close it?

[camalot]

Yeah, closing it is fine by me. the "idea" is still here 😄