Avoid labels containing UIDs in Prometheus Metrics

[RalfWenzel]

The Problem

1. In general, metrics with high cardinality should be avoided if possible.
2. But, as an example, the execution_uri label in the testkube_test_executions_count metrics contains a new uid for each test executed, producing a VERY high cardinality.
3. Besides the cardinality problem, the counter does not work as intended (i guess). The metric value is always = 1 instead of being increased because a new key/value pair is created for every execution -> A new timeseries is created for every execution with value always = 1.
4. Also, the size of the metrics to be scraped increases indefinitely, a new line is added with each test-execution, what makes the metrics practically unusable in production.

Additional context

Below are two lines of metrics as currently provided via the /metrics endpoint of the api-server. Same test (my-test), same result (passed). Each line with value = 1.

testkube_test_executions_count{execution_uri="/tests/my-test/executions/65b90465cf1e8c0df60304a1",labels="executor=curl-executor,test-type=curl-test,webhook=testkube2splunk",name="my-test",result="passed",test_uri="/tests/my-test",type="curl/test"} 1
testkube_test_executions_count{execution_uri="/tests/my-test/executions/65b90465cf1e8c0df60705a1",labels="executor=curl-executor,test-type=curl-test,webhook=testkube2splunk",name="my-test",result="passed",test_uri="/tests/my-test",type="curl/test"} 1

How to improve
I would expect a single line in the metrics with a value of 2. This could be achieved by removing the execution_uri label, or, more broadly spoken, by avoiding UIDs in labels. Example:

testkube_test_executions_count{labels="executor=curl-executor,test-type=curl-test,webhook=testkube2splunk",name="my-test",result="passed",test_uri="/tests/my-test",type="curl/test"} 2

[vsukhin]

hey @RalfWenzel good point, we will fix it