Prometheus-proxy not working for me
nairprakash opened this issue · 10 comments
I have setup an agent and a proxy but agent is not able to communicate to the proxy. Between agent and proxy firewall is open for port 8080, verified it with running nginx on port 8080. Any help in correcting the configuration files would be very much appreciated.
agent:
16:15:00.512 INFO [Agent.kt:171] - Waited 3.00s to reconnect [Agent Unnamed-monitoring]
16:15:00.512 INFO [AgentGrpcService.kt:142] - Connecting to proxy at :8080 using plaintext... [Agent Unnamed-monitoring]
16:15:00.513 INFO [AgentGrpcService.kt:149] - Cannot connect to proxy at :8080 using plaintext - StatusRuntimeException: UNAVAILABLE: Network close
d for unknown reason [Agent Unnamed-monitoring]
Proxy:
16:13:56.243 INFO [Proxy.kt:219] - Version: 1.6.3 Release Date: 12/21/19 [main]
16:13:56.480 INFO [GrpcDsl.kt:84] - Listening for gRPC traffic on port 50051 using plaintext [main]
16:13:56.712 INFO [GenericService.kt:183] - Adding service ProxyGrpcService{serverType=Netty, port=50051} [main]
16:13:56.712 INFO [GenericService.kt:183] - Adding service ProxyHttpService{port=8080} [main]
16:13:56.747 INFO [Log.java:169] - Logging initialized @1163ms to org.eclipse.jetty.util.log.Slf4jLog [main]
16:13:56.842 INFO [GenericService.kt:183] - Adding service AdminService{port=8092, paths=[/ping, /version, /healthcheck, /threaddump]} [main]
16:13:56.843 INFO [GenericService.kt:103] - Enabling Dropwizard metrics [main]
16:13:56.846 INFO [GenericService.kt:106] - Enabling JMX metrics [main]
16:13:56.849 INFO [GenericService.kt:183] - Adding service MetricsService{port=8082, path=/metrics} [main]
16:13:57.006 INFO [GenericService.kt:126] - Zipkin reporter service disabled [main]
16:13:57.009 INFO [GenericService.kt:183] - Adding service Proxy{proxyPort=8080, adminService=AdminService{port=8092, paths=[/ping, /version, /healthcheck, /thr
eaddump]}, metricsService=MetricsService{port=8082, path=/metrics}} [main]
16:13:57.102 INFO [GenericServiceListener.kt:31] - Starting Proxy{proxyPort=8080, adminService=AdminService{port=8092, paths=[/ping, /version, /healthcheck, /th
readdump]}, metricsService=MetricsService{port=8082, path=/metrics}} [main]
16:13:57.105 INFO [GenericServiceListener.kt:31] - Starting MetricsService{port=8082, path=/metrics} [Proxy]
16:13:57.211 INFO [GenericServiceListener.kt:31] - Starting AdminService{port=8092, paths=[/ping, /version, /healthcheck, /threaddump]} [Proxy]
16:13:57.214 INFO [GenericServiceListener.kt:32] - Running MetricsService{port=8082, path=/metrics} [MetricsService STARTING]
16:13:57.373 INFO [GenericServiceListener.kt:32] - Running AdminService{port=8092, paths=[/ping, /version, /healthcheck, /threaddump]} [AdminService STARTING]
16:13:57.379 INFO [GenericServiceListener.kt:31] - Starting ProxyGrpcService{serverType=Netty, port=50051} [Proxy]
16:13:57.484 INFO [GenericServiceListener.kt:31] - Starting ProxyHttpService{port=8080} [Proxy]
16:13:57.483 INFO [GenericServiceListener.kt:32] - Running ProxyGrpcService{serverType=Netty, port=50051} [ProxyGrpcService STARTING]
16:13:57.496 INFO [GenericService.kt:183] - Adding service AgentContextCleanupService{max inactivity secs=15, pause secs=10} [Proxy]
16:13:57.494 INFO [GenericServiceListener.kt:32] - Running ProxyHttpService{port=8080} [ProxyHttpService STARTING]
16:13:57.497 INFO [GenericServiceListener.kt:31] - Starting AgentContextCleanupService{max inactivity secs=15, pause secs=10} [Proxy]
16:13:57.498 INFO [GenericServiceListener.kt:32] - Running Proxy{proxyPort=8080, adminService=AdminService{port=8092, paths=[/ping, /version, /healthcheck, /thr
eaddump]}, metricsService=MetricsService{port=8082, path=/metrics}} [Proxy]
16:13:57.499 INFO [GenericService.kt:138] - All Proxy services healthy [Proxy]
16:13:57.498 INFO [GenericServiceListener.kt:32] - Running AgentContextCleanupService{max inactivity secs=15, pause secs=10} [AgentContextCleanupService]
agent command line : java -jar prometheus-agent.jar -Dagent.proxy.hostname=168.61.46.213:8080 --config /opt/promproxy/apps.conf
proxy docker:
docker run -d --name promproxy -p 8082:8082 -p 8092:8092 -p 50051:50051 -p 8080:8080
--env ADMIN_ENABLED=true
--env METRICS_ENABLED=true
pambrose/prometheus-agent:1.6.3
apps.conf
agent {
pathConfigs: [
{
name: sandboxvm
path: sandboxvm_metrics
url: "http://localhost:9100/metrics"
},
{
name: app2
path: app2_metrics
url: "http://app2.local:9100/metrics"
},
{
name: app3
path: app3_metrics
url: "http://app3.local:9100/metrics"
}
]
}
Can you first verify that you can get the client to reach the proxy locally, without a firewall.
Can you first verify that you can get the client to reach the proxy locally, without a firewall.
Yes, it is working locally.
agent.conf
agent {
proxy { // this part is overrided by the PROXY_HOSTNAME env var
hostname = "10.20.2.12"
port = 50440
}
pathConfigs: [
{
name: testpp
path: testppmetrics
url: "http://localhost:9100/metrics"
}
]
}
Proxy.conf
proxy {
agent.port = 50440
admin.enabled: true
metrics.enabled: true
}
Your log above says the agent is trying to connect to the proxy on port 8080, but your proxy is listening on 50051. Can you see if the agent and proxy or using a common port.
That was my previous attempt.
Keeping the current setting moved the agent.conf to a cloud instance, modified the proxy sever IP to connect to the Application gateway which has port 80 listening for gRPC traffic and it redirects to 10.20.2.12: 50440
Now getting following error:
18:34:18.207 INFO [AgentGrpcService.kt:149] - Cannot connect to proxy at :80 using plaintext - StatusRuntimeException: INTERNAL: http2 exception
The agent uses a gRPC connection to talk to the proxy. gRPC uses HTTP/2, so the connection between your agent needs to support HTTP/2. Can you have the proxy not sit behind anything?
This communication needs to happen between two different cloud subscription.
Ah. The problem is not all proxies support HTTP/2.
Http2 is enabled on the proxy IP:port and nginx internally sends it to 10.20.2.12: 50440.
grpc_pass grpc://localhost:50440;
nginx is on board with HTTP/2: https://www.nginx.com/blog/nginx-1-13-10-grpc/
I will give it a try and see how it works. It will be a couple days before I can get to it though.
thanks