Forwarding match[] params
Closed this issue · 18 comments
Hi
I'm trying out the proxy for federating multiple Prometheus instances across network boundaries.
I have got a fairly simple POC up and running locally looking something like this
My question is does the proxy support the forwarding of queries to the federate endpoint?
To support the question I have been playing about with the following config.
If I configure my agent like this, with just the federate endpoint configured it doesn't seem to pass through the query param from the prometheus config. I have even tried accessing promprox-proxy:8080/local_prom?match[]={__name__=~'..*'} in the browser and i get no metrics.
agent {
pathConfigs: [
{
name: local_prom_metrics
path: local_prom
url: "http://prometheus:9090/federate"
}
]
}
scrape_configs:
- job_name: 'proxied-metrics'
params:
'match[]':
- '{__name__=~"..*"}'
metrics_path: '/local_prom'
static_configs:
- targets: ['promprox-proxy:8080']But if I change the agent to this (or some variant of)
agent {
pathConfigs: [
{
name: local_prom_metrics
path: local_prom
url: "http://prometheus:9090/federate?match[]={__name__=~'..*'}"
}
]
}
I then begin to get metrics through the proxy (without changing the prom config at all)
Hi Scott,
I think I understand what you are trying to do, but I am not sure. Let me offer these comments and you can tell me if they are helpful. I should also add that I have not dealt with a Prometheus federation, so my ignorance may bleed through.
-
The pathConfig url is used literally and does support the regex-like syntax that you are using.
-
The pathConfig url ordinarily points at a server with metrics. In the diagram above, you have it querying Prometheus. Is it pulling metrics derived from other endpoints with that connection?
-
I am open to the idea of supporting the syntax you used above, but I need a little context for it.
Cheers,
Paul
Hi Paul
When you query the /federation endpoint of a Prometheus server it can expose metrics in regular Prometheus format. But by default it does not give back anything, this is to prevent exposing all metrics by default, which would be a lot of data in some instances.
So to enable the federated endpoint to return any metrics you have to send it a query string param(s) to tell it what you want. In this case, I want everything (for testing) and use the query {__name__=~'..*'}.
What I would like is not to have to specify this in the path config in the agent (which would be behind the firewall and need some effort to change) but to be able to append that query string to the request to the proxy and have it forwarded through to the agent and eventually the target that is being scraped.
In this way, I can control what is being scraped from the federation endpoint from the configuration of the Prometheus instance that is scraping the proxy. I can then tune what metrics I want from anything that is being proxied by changing this configuration centrally.
Okay, I think I understand.
The proxy does not need to do anything special with the regex args, you just want to be able to specify args dynamically on the proxy side and pass them to the agent on each request.
Is that correct?
Would it be okay to specify them as a query param value as part of the URL specified in the prometheus.yml file? Or are you thinking as part of the proxy config file.
One problem doing it with query params is the URL encoding could get ugly if the user specifies it the prometheus.yml file. If we do it in the proxy config file, we can do the encoding programmatically.
How about a hybrid of the two config files.
In the prometheus.yml you file would add a query param value:
- targets: ['mymachine.local:8080?match=config1']
Then in the proxy.conf file, you would map the name used in the query param to a value:
proxy {
queryConfigs: [
name: "config1"
arg: "{name=~'..*'}"
]
}
That would keep you out of the URL encoding business and you would not have to adjust the agent configs each time.
Is a single value like "{name=~'..*'}" sufficient for the desired query?
Hi
you dont add the params directly to the target URL in the prometheus.yml you add them in the params property
scrape_configs:
- job_name: 'proxied-metrics'
params:
'match[]':
- '{__name__=~"..*"}'
metrics_path: '/local_prom'
static_configs:
- targets: ['promprox-proxy:8080']
They are then picked up by Prometheus and added to the URL that is scraped by Prometheus.
In this case, the URL would be the proxy URL and it would have params added to the URL being scraped. They are calculated and encoded by Prometheus before being added to the scrape URL. These params should be "passed through" to the target being proxied.
It might be my unfamiliarity with the code but I can't see why you would want/need to configure the params upfront in the proxy/agent config
The params section in the prometheus.yml is a standard part of the config and should handle any encoding issues with the params that are added there.
I think I am lost in the configs, URLs, and queries being discussed. Let me ask the question another way: in an ideal world, how would you like to see the proxy config files altered to accommodate the your desired query?
Yup it's getting quite confusing!!
I don't think the config needs changing at all.
I think that query params are getting stripped off by the proxy.
I would like it to forward query params to the target being scraped.
Ah, I think I get it now! That does not sound hard. Let me look at it later today.
I had a scan through the code and it looks like it just drops query params off. I'm on my phone so can't find the exact lines.
It was an easy fix.
Give this a try and see what you think:
https://github.com/pambrose/prometheus-proxy/releases/tag/1.6.4b1
Great. I had been using the docker containers can you push new ones with this code? or is there another way to get containers built?
OK i can see you just copy the jar into the container so i think i can modify my dockerfiles to copy this new jar over the top of the old version
That seems to have sorted the issue!
The query params are being passed like this
Prometheus -> prom-proxy -> prom-agent -> target
So now on the "external" Prometheus, I can vary the query params to be able to scrape different values from the "target" Prometheus
Great!
Sorry for not generating the docker image!
I will kick out a new release today with the docker releases.
I pushed out the 1.6.4 release, including the docker images. Let me know if you run into any problems with it.
Thanks for the kind words.