Multiple Dynamic Label mapping in external probe

Describe the feature you'd like and the problem it will solve
Right now With external probe we can map only single dynamic labels and later we can add multiple static labels using prober config.So this allows only to define metrics like
db_stats{name=db01,ptype="external",probe="external_test"} 1.0

However we would like to have multiple dynamic label mapping like below

db_stats{name="db01",region="us-west-2",status="running",ptype="external",probe="external_test"} 1.0
db_stats{name="db02",region="us-east-1",status="running",ptype="external",probe="external_test"} 1.0
db_stats{name="db03",region="eu-west-1",status="stopped",ptype="external",probe="external_test"} 1.0

This feature can certainly allow cloudprober as an alternative to Blackbox Exporter+Push Gateway and makes external probes more useful.

The same issue was discussed in https://cloudprober.slack.com/archives/C9ZPA5611/p1573234901001700 and @manugarg provided some workaround to use prometheus relabel but this does not scale well if we are looking at more(~5 or more) different labels and we have more and more use case for the same scenario.It would be great if we can add this feature for the external probe

Just to make sure I am understanding it correctly, the ask here is to allow external program to specify labels in the payload itself. Is that correct?

So, the external program will create a payload like this:
db_stats{name="db01",region="us-west-2",status="running"} 1.0

And, cloudprober will then collate these labels with the internal labels and final metric will look like this:
db_stats{name="db01",region="us-west-2",status="running",ptype="external",probe="external_test"} 1.0

Can you please confirm?

Also, there is an undocumented feature of cloudprober, wherein you can add additional labels to the probe in the probe config:
https://github.com/google/cloudprober/blob/master/examples/additional_label/cloudprober.cfg

These labels can be based on some metadata of the instance where probe is running and target labels. For example, if target is a discovered target and has the label "region", you can add that label's value to additional labels like this:

additional_label {
  key: "region"
  value: "target.label.region"
}

I am guessing this won't be sufficient for you, please let me know.

@manugarg : Additional label is kind of static label that gets attached to all the metrics generated from that probe.This is useful when we need to segregate metrics based on common labels like environment or region etc.

What we are looking here is on the dynamic label.You understood it correctly as mentioned above.The external program creates the payload
db_stats{name="db01",region="us-west-2",status="running"} 1.0

And, cloudprober will then collate the labels with the internal labels and final metric should look like :
db_stats{name="db01",region="us-west-2",status="running",ptype="external",probe="external_test"} 1.0

@infa-kparida Thanks for confirming. Just wanted to let you know that it will take a bit longer to fix, because of how payload metrics are handled: we keep one set of metrics (called EventMetrics internally) where they all have the same labels. We can solve this by handling each metric individually, but that will be incompatible (or hard to implement) with the "aggregate_in_cloudprober" option. I guess that's okay, but we'll have to look at it more closely.

I'll have more info soon'ish.

Additional label is kind of static label that gets attached to all the metrics generated from that probe.This is useful when we need to segregate metrics based on common labels like environment or region etc.

Just wanted to let you know that additional labels can also be set based on the target labels, so they are not that static. I updated the example here:

cloudprober/examples/additional_label/cloudprober.cfg

Line 48 in 03e7328

value: "@target.label.fqdn@"

Not saying that they solve this issue.

Thanks @manugarg for taking this up.Its not a blocker for us but just a good feature to have.So we are fine if it takes longer time :)

@infa-kparida #486 implements dynamic labels. Whenever you get a chance, you please verify if that's what you were looking for?

@manugarg : This adds label to any new metrics that are being emitted from prober. Is there a way to add labels to standard metrics like (total, success)