Question: Is TLS encryption supported ?

Question

Question: Is TLS encryption supported ?

Altiire opened this issue 5 years ago · 28 comments

Hi,
I need to push my data outside of a firewall to a server based in the cloud.
So my data needs to be encrypted. I saw that gRPC can use TLS encryption on the client side and I wanted to know if you support it.

Thanks,

Source: https://www.nginx.com/blog/nginx-1-13-10-grpc/

Answer 1 · 2019-11-28T16:33:11.000Z

Hi Raphaël,
Yes, I need to add TLS support. Give me a few days and I will get back to you with it.
Thanks,
Paul

Answer 2 · 2019-12-05T08:03:56.000Z

Hi Raphaël,

I added TLS support for the gRPC connections in the 1.5.0 release. It supports running with and without mutual authentication. Let me know how it works for you.

Thanks,
Paul

Answer 3 · 2019-12-05T09:55:53.000Z

Hi Paul,

Thank you very much !
I have a few things to do but I will keep you in touch as soon as possible.

Regards,

Answer 4 · 2019-12-09T16:53:45.000Z

Hi Paul,

Quick follow up, my proxy is behind nginx-proxy and let's encrypt companion.

I've got the following log on the agent but nothing on my proxy:

16:43:30.280 INFO  [AgentGrpcService.kt:137] - Connecting to proxy at myproxy.example.com:443 using TLS (no mutual auth)... [Agent Unnamed-d2acdfcd6bb7]
16:43:30.281 INFO  [AgentGrpcService.kt:146] - Cannot connect to proxy at myproxy.example.com:443 using TLS (no mutual auth) - StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0] [Agent Unnamed-d2acdfcd6bb7]

It looks like that my agent cannot connect to my proxy but it does not actually try to.
Perhaps it is because that nginx-proxy is waiting for https ? Or, seeing the timestamp, something fail before ?

I'm on the alpine tag for nginx-proxy for http2 support so grpcs can be encapsulated to https and proxy-ed by nginx.

Regards,

Answer 5 · 2019-12-09T16:59:29.000Z

The log says you are trying to connect to myproxy.example.com, which suggests you have not provided your own proxy hostname to your agent. You can do that via the config file, ENV VAR or CLI arg. Are you running the agent as an uber jar or docker container?

Answer 6 · 2019-12-09T17:02:05.000Z

I replaced the real name of my proxy :)

I'm running the agent as a docker container, on a remote host (that's why I'm using nginx-proxy with let's encrypt). My proxy is also running as a docker container and in the same network as my ngnix-proxy.

Answer 7 · 2019-12-09T17:05:38.000Z

Ah! :-)

In the interest of making sure that base elements are working and isolating the issue, can you try connecting to the proxy directly , without nginx in the middle? I have not tried using nginx with let's encrypt. I will give it a try on my end.

Answer 8 · 2019-12-09T17:08:30.000Z

Well, everything is in place (the certs and configs) so I will try to expose my proxy tomorrow morning. I keep you in touch ASAP.

Regards,

Answer 9 · 2019-12-11T09:51:38.000Z

Hi Paul,

Sorry for the delay.

So I opened 50440/tcp port on the proxy and the agent connect directly to the proxy but it is not working. The logs are as follow:

on the proxy:

09:42:28.471 ERROR [ProxyTransportFilter.kt:44] - Null attributes [grpc-nio-worker-ELG-3-1]

on the agent:

09:50:01.038 INFO  [AgentGrpcService.kt:137] - Connecting to proxy at myproxy.example.com:50440 using TLS (no mutual auth)... [Agent Unnamed-c58dcca080d1]
09:50:01.038 INFO  [AgentGrpcService.kt:146] - Cannot connect to proxy at myproxy.example.com:50440 using TLS (no mutual auth) - StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0] [Agent Unnamed-c58dcca080d1]

If you have any idea,
Regards,

Answer 10 · 2019-12-12T03:49:50.000Z

Hi Raphaël,

In the interest of first getting to a working, known state, can you try using the certificates, keys and configs present in the repo (https://github.com/pambrose/prometheus-proxy/blob/master/examples/tls-no-mutual-auth.conf), with both the agent and proxy running on your local machine?

After that, try moving the agent and proxy to a different machines.

After that try it with your own certificate and keys.

I have seen the "Null attributes" error in the log before and it was always a configuration issue.

Paul

Answer 11 · 2019-12-12T09:27:43.000Z

Hi Paul,

So, I cloned the repo and I had to adapt you documentation to get it working:

docker network create prometheus-proxy

docker run --rm -p 8082:8082 -p 8092:8092 -p 50440:50440 -p 8080:8080 \
    --mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
    --mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
    --env PROXY_CONFIG=tls-no-mutual-auth.conf \
    --env ADMIN_ENABLED=true \
    --env METRICS_ENABLED=true --name docker-proxy --net=prometheus-proxy\
    pambrose/prometheus-proxy:1.5.0

docker run --rm -p 8083:8083 -p 8093:8093 \
    --mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
    --mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
    --env AGENT_CONFIG=tls-no-mutual-auth.conf \
    --env PROXY_HOSTNAME=docker-proxy:50440 \
    --name docker-agent --net=prometheus-proxy\
    pambrose/prometheus-agent:1.5.0

I will now try using your certificates, keys and configs on my remotes.

For the record, here are my config files:

/etc/prometheus-proxy/prometheus-proxy.conf

proxy {
  agent.port = 50440

  tls {
    certChainFilePath = "/etc/prometheus-proxy/cert.pem"
    privateKeyFilePath = "/etc/prometheus-proxy/key.pem"
    trustCertCollectionFilePath = ""
  }
}

/etc/prometheus-agent/prometheus-agent.conf

agent {
  proxy { // this part is overrided by the PROXY_HOSTNAME env var
    hostname = "myproxy.example.com"
    port = 443
  }

  tls {
    certChainFilePath = ""
    privateKeyFilePath = ""
    trustCertCollectionFilePath = "/etc/prometheus-agent/ca.pem"
  }

  pathConfigs: [
    {
      name: "service.example.com cadvisor"
      path: "service.example.com_cadvisor_metrics"
      url: "cadvisor:8080/metrics"
    },
    {
      name: "service.example.com node_exporter"
      path: "service.example.com_node_exporter_metrics"
      url: "node_exporter:9100/metrics"
    }
  ]
}

Answer 12 · 2019-12-12T10:29:49.000Z

The problem seems to be my certificates and keys as it was working with yours and it stopped with mine. You can close the issue :)

Answer 13 · 2019-12-12T11:13:01.000Z

I can't make it work with my letsencrypt certificate/key or self-signed ones, gRPC needs special certificates/keys ?

Answer 14 · 2019-12-12T13:02:40.000Z

Back with some news, I am able to use my letsencrypt certificate/key if I use port 443 and root user.
So my proxy is running without errors and my agent still cannot connect. There is no event on my nginx-proxy listening on port 443.

Answer 15 · 2019-12-12T14:51:22.000Z

The code that actually deals with the TLS specifics is buried in the gRPC code, so I am at a disadvantage trying to make sense of what you are seeing. The gRPC docs mention generating your own self-signed certificates with commands in the test certs README. Can you try your same setup with self-signed certificates and see if that works. If self-signed certs work, but your letsencrypt certs do not, then we can ask the gRPC folks about the issue.

Answer 16 · 2019-12-12T14:55:20.000Z

I will try with the documentation you gave me. But the proxy is working with user root on port 443. It's weird that my nginx-proxy does not show any request from the agent no ?

Answer 17 · 2019-12-12T14:58:41.000Z

Agreed. Can you verify that the agent is using a --env PROXY_HOSTNAME=docker-proxy:50440 value in the docker agent invocation that points to nginx-proxy:443.

Answer 18 · 2019-12-12T15:01:41.000Z

Not anymore, it is now pointing to the remote machine on port 443. It worked when it was pointing to the remote on port 50440 with your certificates/keys. As said before, to make the proxy work with my certificates/keys, I had to use the port 443.

Answer 19 · 2019-12-12T15:05:27.000Z

Sorry, just so I am clear, it works when you use the repo certs on port 50440 and the nginx-proxy is not involved. But when you add the nginx-proxy and letsencrypt certs, it does not work on port 443.

Answer 20 · 2019-12-12T15:09:25.000Z

it works when you use the repo certs on port 50440 and the nginx-proxy is not involved

yes

But when you add the nginx-proxy and letsencrypt certs, it does not work on port 443

kind of, to be able to use the letsencrypt certs, I need to use the port 443 (otherwise, the proxy has a null pointer exception). To be able to use the port 443 with my agent, I need to add nginx-proxy to the equation.
I will try to run the agent on the same machine with the port 443 (to by pass nginx-proxy), I keep you up to date ASAP

Answer 21 · 2019-12-12T15:29:15.000Z

Given nginx clearly supports gRPC in their docs, I think we will eventually get this to work.

Answer 22 · 2019-12-16T14:37:48.000Z

Hi Paul,

Back with some news.

My agent is on the same remote machine than my proxy. They were using a self-signed certificate for "prometheus-proxy", the name of my container as they are in the same docker network. And it was working.

I then generated a certificate for prometheus-proxy.example.com with prometheus-proxy.example.com as the name of my container. And it was working.

I put my proxy is back again behind the first nginx, the one that makes a grpc_pass using the nginx:stable-alpine image. Initially, I wanted it to make a grpc_pass with the grpcs proto.
I could not make it working so I made something different:
agent is using tls to connect to the nginx, the nginx make a grpc_pass with the grpc proto to port 50440 without using tls, so the agent is using tls but not my proxy as i want to have a secure data transfert when i m on the internet.
I now have a new error:

on the proxy:

14:29:21.934 INFO  [ProxyServiceImpl.kt:73] - registerAgent() missing AgentContext agentId: 269 [grpc-default-executor-0]
14:29:21.936 INFO  [ProxyTransportFilter.kt:50] - Disconnected from AgentContext{agentId=270, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=2.55ms} [grpc-nio-worker-ELG-3-2]

on the agent:

14:30:03.906 INFO  [AgentGrpcService.kt:137] - Connecting to proxy at nginx-prometheus-proxy:443 using TLS (no mutual auth)... [Agent Unnamed-3cd7dfe83c04]
14:30:03.946 INFO  [AgentGrpcService.kt:139] - Connected to proxy at nginx-prometheus-proxy:443 using TLS (no mutual auth) [Agent Unnamed-3cd7dfe83c04]
14:30:03.955 INFO  [Agent.kt:115] - Disconnected from proxy at nginx-prometheus-proxy:443 after invalid response registerAgent() - Invalid agentId: 297 [Agent Unnamed-3cd7dfe83c04]

My actual configuration is a self-signed certificate for nginx-prometheus-proxy, agent connecting to nginx-prometheus-proxy and nginx-prometheus-proxy forwarding request using grpc_pass on the grpc proto (so plain text).
Do the agent support to use tls but not the proxy ? Or do you have another idea ?

Answer 23 · 2019-12-17T03:00:20.000Z

Hi Raphaël,

As I am sure you have seen, nginx claims to work fine with gRPC. Let me do some more research on the topic. I will get back to you later this week. In the meantime, can you check to see that your setup matches the example configuration provided in that post.

Paul

Answer 24 · 2019-12-17T09:11:17.000Z

Hi Paul,

You think the logs bellow are a nginx problem ? Because they succeed to communicate...

My proxy config:

proxy {
  agent.port = 50440

//  tls {
//    certChainFilePath = "/etc/prometheus-proxy/server.pem"
//    privateKeyFilePath = "/etc/prometheus-proxy/server.key"
//    trustCertCollectionFilePath = ""
//  }
}

my agent config:

agent {
  proxy {
    hostname = "prometheus-proxy.myexample.com"
    port = 50440
  }

  tls {
    certChainFilePath = ""
    privateKeyFilePath = ""
    trustCertCollectionFilePath = "/etc/prometheus-agent/ca.pem"
  }

  pathConfigs: [
    {
      name: "prometheus cadvisor"
      path: "prometheus_cadvisor_metrics"
      url: "cadvisor:8080/metrics"
    },
    {
      name: "prometheus node_exporter"
      path: "prometheus_node_exporter_metrics"
      url: "node_exporter:9100/metrics"
    }
  ]
}

with hostname and port overrided by the env var.

Answer 25 · 2019-12-27T11:05:02.000Z

Hi @pambrose ,
Do you have any news ? I upgraded to 1.6.3

Answer 26 · 2019-12-27T20:04:09.000Z

Hi Raphaël,
I have been busy with something else. Give me another week to look into it.
Paul

Answer 27 · 2019-12-30T16:04:21.000Z

Hi Paul,
I have a few logs for you that may help you.
The two first are for the proxy and agent with nginx in the middle (with no SSL).
The other two are for the proxy and agent without nginx.
I think that there is a problem with the agentId when an error occur... So there are 2 problems as there is an error while using nginx.

I made a test with https://github.com/grpc/grpc/tree/master/examples/python/data_transmission and https://github.com/grpc/grpc/tree/master/examples/python/helloworld using the nginx:stable-alpine image without error so I'm sure that nginx is not the problem.

Proxy with nginx:

16:27:11.713 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] OUTBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192} [grpc-nio-worker-ELG-3-2]
16:27:11.714 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041 [grpc-nio-worker-ELG-3-2]
16:27:11.721 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] INBOUND SETTINGS: ack=false settings={HEADER_TABLE_SIZE=0, ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=2147483647} [grpc-nio-worker-ELG-3-2]
16:27:11.721 DEBUG [AbstractInternalLogger.java:193] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] OUTBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-3-2]
16:27:11.722 DEBUG [ProxyTransportFilter.kt:34] - Registering agentId: 9 [grpc-nio-worker-ELG-3-2]
16:27:11.722 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=2147418112 [grpc-nio-worker-ELG-3-2]
16:27:11.723 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] INBOUND HEADERS: streamId=1 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/connectAgent, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] padding=0 endStream=false [grpc-nio-worker-ELG-3-2]
16:27:11.723 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] INBOUND DATA: streamId=1 padding=0 endStream=true length=5 bytes=0000000000 [grpc-nio-worker-ELG-3-2]
16:27:11.724 DEBUG [AbstractInternalLogger.java:193] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] INBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-3-2]
16:27:11.726 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] OUTBOUND HEADERS: streamId=1 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, agent-id: 9, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-2]
16:27:11.727 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] OUTBOUND DATA: streamId=1 padding=0 endStream=false length=5 bytes=0000000000 [grpc-nio-worker-ELG-3-2]
16:27:11.727 DEBUG [AbstractInternalLogger.java:216] - [id: 0x4c74fc6a, L:/172.25.0.4:50440 - R:/172.25.0.3:52916] OUTBOUND HEADERS: streamId=1 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-3-2]
16:27:11.736 DEBUG [Proxy.kt:183] - Removed AgentContext{agentId=9, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=14.2ms} [grpc-nio-worker-ELG-3-2]
16:27:11.736 INFO  [ProxyTransportFilter.kt:50] - Disconnected from AgentContext{agentId=9, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=14.5ms} [grpc-nio-worker-ELG-3-2]
16:27:11.738 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] OUTBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192} [grpc-nio-worker-ELG-3-3]
16:27:11.738 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041 [grpc-nio-worker-ELG-3-3]
16:27:11.740 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] INBOUND SETTINGS: ack=false settings={HEADER_TABLE_SIZE=0, ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=2147483647} [grpc-nio-worker-ELG-3-3]
16:27:11.741 DEBUG [AbstractInternalLogger.java:193] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] OUTBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-3-3]
16:27:11.741 DEBUG [ProxyTransportFilter.kt:34] - Registering agentId: 10 [grpc-nio-worker-ELG-3-3]
16:27:11.741 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=2147418112 [grpc-nio-worker-ELG-3-3]
16:27:11.742 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] INBOUND HEADERS: streamId=1 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/registerAgent, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] padding=0 endStream=false [grpc-nio-worker-ELG-3-3]
16:27:11.743 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] INBOUND DATA: streamId=1 padding=0 endStream=true length=43 bytes=00000000260a01391214556e6e616d65642d3866333762306534346332361a0b6e67696e782d70726f7879 [grpc-nio-worker-ELG-3-3]
16:27:11.743 DEBUG [AbstractInternalLogger.java:193] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] INBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-3-3]
16:27:11.744 INFO  [ProxyServiceImpl.kt:74] - registerAgent() missing AgentContext agentId: 9 [grpc-default-executor-0]
16:27:11.745 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] OUTBOUND HEADERS: streamId=1 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, agent-id: 10, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-3]
16:27:11.746 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] OUTBOUND DATA: streamId=1 padding=0 endStream=false length=28 bytes=00000000171212496e76616c6964206167656e7449643a20391a0139 [grpc-nio-worker-ELG-3-3]
16:27:11.746 DEBUG [AbstractInternalLogger.java:216] - [id: 0x909ba5b4, L:/172.25.0.4:50440 - R:/172.25.0.3:52918] OUTBOUND HEADERS: streamId=1 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-3-3]
16:27:11.752 DEBUG [Proxy.kt:183] - Removed AgentContext{agentId=10, valid=true, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=11.3ms} [grpc-nio-worker-ELG-3-3]
16:27:11.753 INFO  [ProxyTransportFilter.kt:50] - Disconnected from AgentContext{agentId=10, valid=false, agentName=Unassigned, hostName=Unassigned, remoteAddr=Unknown, lastRequestDuration=11.6ms} [grpc-nio-worker-ELG-3-3]
^C*** Proxy shutting down ***
16:27:14.393 INFO  [GenericServiceListener.kt:33] - Stopping Proxy{proxyPort=8080, adminService=AdminService{port=8092, paths=[/ping, /version, /healthcheck, /threaddump, /debug]}, metricsService=MetricsService{port=8082, path=/metrics}} [Thread-17]

Agent with nginx:

16:27:11.686 INFO  [Agent.kt:171] - Waited 3.00s to reconnect [Agent Unnamed-8f37b0e44c26]
16:27:11.687 INFO  [AgentGrpcService.kt:116] - Creating gRPC stubs [Agent Unnamed-8f37b0e44c26]
16:27:11.687 INFO  [GrpcDsl.kt:50] - Creating connection for gRPC server at nginx-proxy:443 using plaintext [Agent Unnamed-8f37b0e44c26]
16:27:11.688 INFO  [Agent.kt:128] - Resetting agentId [Agent Unnamed-8f37b0e44c26]
16:27:11.688 INFO  [AgentGrpcService.kt:142] - Connecting to proxy at nginx-proxy:443 using plaintext... [Agent Unnamed-8f37b0e44c26]
16:27:11.688 DEBUG [AbstractInternalLogger.java:216] - [id: 0x6ba458ee, L:/172.25.0.2:50086 - R:nginx-proxy/172.25.0.3:443] OUTBOUND GO_AWAY: lastStreamId=0 errorCode=0 length=0 bytes= [grpc-nio-worker-ELG-1-2]
16:27:11.696 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192} [grpc-nio-worker-ELG-1-4]
16:27:11.706 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041 [grpc-nio-worker-ELG-1-4]
16:27:11.707 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=128, INITIAL_WINDOW_SIZE=65536, MAX_FRAME_SIZE=16777215} [grpc-nio-worker-ELG-1-4]
16:27:11.707 DEBUG [AbstractInternalLogger.java:193] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] OUTBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-1-4]
16:27:11.708 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=2147418112 [grpc-nio-worker-ELG-1-4]
16:27:11.708 DEBUG [AbstractInternalLogger.java:193] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-1-4]
16:27:11.709 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:authority: nginx-proxy:443, :path: /ProxyService/connectAgent, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-4]
16:27:11.710 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] OUTBOUND DATA: streamId=3 padding=0 endStream=true length=5 bytes=0000000000 [grpc-nio-worker-ELG-1-4]
16:27:11.729 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[:status: 200, server: nginx/1.16.1, date: Mon, 30 Dec 2019 15:27:11 GMT, content-type: application/grpc, agent-id: 9, grpc-encoding: identity, grpc-accept-encoding: gzip] padding=0 endStream=false [grpc-nio-worker-ELG-1-4]
16:27:11.730 DEBUG [AgentClientInterceptor.kt:57] - Assigned agentId to Agent{agentId=9, agentName=Unnamed-8f37b0e44c26, proxyHost=nginx-proxy:443, adminService=AdminService{port=8093, paths=[/ping, /version, /healthcheck, /threaddump, /debug]}, metricsService=Disabled} [Agent Unnamed-8f37b0e44c26]
16:27:11.730 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND DATA: streamId=3 padding=0 endStream=false length=5 bytes=0000000000 [grpc-nio-worker-ELG-1-4]
16:27:11.731 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] padding=0 endStream=true [grpc-nio-worker-ELG-1-4]
16:27:11.731 INFO  [AgentGrpcService.kt:144] - Connected to proxy at nginx-proxy:443 using plaintext [Agent Unnamed-8f37b0e44c26]
16:27:11.733 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:authority: nginx-proxy:443, :path: /ProxyService/registerAgent, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-4]
16:27:11.733 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] OUTBOUND DATA: streamId=5 padding=0 endStream=true length=43 bytes=00000000260a01391214556e6e616d65642d3866333762306534346332361a0b6e67696e782d70726f7879 [grpc-nio-worker-ELG-1-4]
16:27:11.747 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[:status: 200, server: nginx/1.16.1, date: Mon, 30 Dec 2019 15:27:11 GMT, content-type: application/grpc, agent-id: 10, grpc-encoding: identity, grpc-accept-encoding: gzip] padding=0 endStream=false [grpc-nio-worker-ELG-1-4]
16:27:11.748 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND DATA: streamId=5 padding=0 endStream=false length=28 bytes=00000000171212496e76616c6964206167656e7449643a20391a0139 [grpc-nio-worker-ELG-1-4]
16:27:11.749 DEBUG [AbstractInternalLogger.java:216] - [id: 0x750a2cfd, L:/172.25.0.2:50092 - R:nginx-proxy/172.25.0.3:443] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] padding=0 endStream=true [grpc-nio-worker-ELG-1-4]
16:27:11.750 INFO  [Agent.kt:164] - Disconnected from proxy at nginx-proxy:443 after invalid response registerAgent() - Invalid agentId: 9 [Agent Unnamed-8f37b0e44c26]

Proxy without nginx:

16:34:19.577 INFO  [BaseApplicationEngine.kt:39] - Responding at http://0.0.0.0:8080 [DefaultDispatcher-worker-1]
16:34:25.460 DEBUG [AbstractByteBuf.java:62] - -Dio.netty.buffer.checkAccessible: true [grpc-nio-worker-ELG-3-1]
16:34:25.461 DEBUG [AbstractByteBuf.java:63] - -Dio.netty.buffer.checkBounds: true [grpc-nio-worker-ELG-3-1]
16:34:25.464 DEBUG [ResourceLeakDetectorFactory.java:195] - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@175808bf [grpc-nio-worker-ELG-3-1]
16:34:25.574 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192} [grpc-nio-worker-ELG-3-1]
16:34:25.580 DEBUG [Recycler.java:96] - -Dio.netty.recycler.maxCapacityPerThread: 4096 [grpc-nio-worker-ELG-3-1]
16:34:25.581 DEBUG [Recycler.java:97] - -Dio.netty.recycler.maxSharedCapacityFactor: 2 [grpc-nio-worker-ELG-3-1]
16:34:25.581 DEBUG [Recycler.java:98] - -Dio.netty.recycler.linkCapacity: 16 [grpc-nio-worker-ELG-3-1]
16:34:25.581 DEBUG [Recycler.java:99] - -Dio.netty.recycler.ratio: 8 [grpc-nio-worker-ELG-3-1]
16:34:25.599 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041 [grpc-nio-worker-ELG-3-1]
16:34:25.614 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192} [grpc-nio-worker-ELG-3-1]
16:34:25.616 DEBUG [AbstractInternalLogger.java:193] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-3-1]
16:34:25.629 DEBUG [ProxyTransportFilter.kt:34] - Registering agentId: 1 [grpc-nio-worker-ELG-3-1]
16:34:25.632 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041 [grpc-nio-worker-ELG-3-1]
16:34:25.633 DEBUG [AbstractInternalLogger.java:193] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-3-1]
16:34:25.766 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND HEADERS: streamId=3 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/connectAgent, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:25.815 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND DATA: streamId=3 padding=0 endStream=true length=5 bytes=0000000000 [grpc-nio-worker-ELG-3-1]
16:34:25.857 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:25.881 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND DATA: streamId=3 padding=0 endStream=false length=5 bytes=0000000000 [grpc-nio-worker-ELG-3-1]
16:34:25.882 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-3-1]
16:34:25.945 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND HEADERS: streamId=5 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/registerAgent, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:25.946 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND DATA: streamId=5 padding=0 endStream=true length=48 bytes=000000002b0a01311214556e6e616d65642d3866333762306534346332361a1070726f6d6574686575732d70726f7879 [grpc-nio-worker-ELG-3-1]
16:34:25.962 INFO  [ProxyServiceImpl.kt:73] - Connected to AgentContext{agentId=1, valid=true, agentName=Unnamed-8f37b0e44c26, hostName=prometheus-proxy, remoteAddr=Unknown, lastRequestDuration=330ms} [grpc-default-executor-0]
16:34:25.969 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:25.979 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND DATA: streamId=5 padding=0 endStream=false length=30 bytes=000000001908011212496e76616c6964206167656e7449643a20311a0131 [grpc-nio-worker-ELG-3-1]
16:34:25.979 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-3-1]
16:34:25.996 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND HEADERS: streamId=7 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/registerPath, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:25.998 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND DATA: streamId=7 padding=0 endStream=true length=51 bytes=000000002e0a013112297b7b20696e76656e746f72795f686f73746e616d65207d7d5f6361647669736f725f6d657472696373 [grpc-nio-worker-ELG-3-1]
16:34:26.001 INFO  [ProxyPathManager.kt:44] - Added path /{{ inventory_hostname }}_cadvisor_metrics for AgentContext{agentId=1, valid=true, agentName=Unnamed-8f37b0e44c26, hostName=prometheus-proxy, remoteAddr=Unknown, lastRequestDuration=372ms} [grpc-default-executor-0]
16:34:26.003 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=7 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:26.005 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND DATA: streamId=7 padding=0 endStream=false length=29 bytes=000000001808011212496e76616c6964206167656e7449643a20311801 [grpc-nio-worker-ELG-3-1]
16:34:26.005 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=7 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-3-1]
16:34:26.015 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND HEADERS: streamId=9 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/registerPath, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:26.018 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND DATA: streamId=9 padding=0 endStream=true length=56 bytes=00000000330a0131122e7b7b20696e76656e746f72795f686f73746e616d65207d7d5f6e6f64655f6578706f727465725f6d657472696373 [grpc-nio-worker-ELG-3-1]
16:34:26.020 INFO  [ProxyPathManager.kt:44] - Added path /{{ inventory_hostname }}_node_exporter_metrics for AgentContext{agentId=1, valid=true, agentName=Unnamed-8f37b0e44c26, hostName=prometheus-proxy, remoteAddr=Unknown, lastRequestDuration=391ms} [grpc-default-executor-0]
16:34:26.021 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=9 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:26.022 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND DATA: streamId=9 padding=0 endStream=false length=31 bytes=000000001a08011212496e76616c6964206167656e7449643a203118022001 [grpc-nio-worker-ELG-3-1]
16:34:26.023 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=9 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-3-1]
16:34:26.048 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND HEADERS: streamId=11 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/readRequestsFromProxy, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:26.050 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND DATA: streamId=11 padding=0 endStream=true length=8 bytes=00000000030a0131 [grpc-nio-worker-ELG-3-1]
16:34:26.138 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND HEADERS: streamId=13 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/writeResponsesToProxy, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:26.148 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND HEADERS: streamId=15 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/writeChunkedResponsesToProxy, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:31.174 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND HEADERS: streamId=17 headers=GrpcHttp2RequestHeaders[:path: /ProxyService/sendHeartBeat, :authority: prometheus-proxy:50440, :method: POST, :scheme: http, te: trailers, content-type: application/grpc, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:31.175 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] INBOUND DATA: streamId=17 padding=0 endStream=true length=8 bytes=00000000030a0131 [grpc-nio-worker-ELG-3-1]
16:34:31.181 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=17 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-3-1]
16:34:31.195 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND DATA: streamId=17 padding=0 endStream=false length=27 bytes=000000001608011212496e76616c6964206167656e7449643a2031 [grpc-nio-worker-ELG-3-1]
16:34:31.196 DEBUG [AbstractInternalLogger.java:216] - [id: 0x25764b9d, L:/172.25.0.4:50440 - R:/172.25.0.2:37058] OUTBOUND HEADERS: streamId=17 headers=GrpcHttp2OutboundHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-3-1]

Agent without nginx:

16:34:24.958 INFO  [AgentGrpcService.kt:142] - Connecting to proxy at prometheus-proxy:50440 using plaintext... [Agent Unnamed-8f37b0e44c26]
16:34:25.191 DEBUG [AbstractByteBuf.java:62] - -Dio.netty.buffer.checkAccessible: true [grpc-default-executor-0]
16:34:25.191 DEBUG [AbstractByteBuf.java:63] - -Dio.netty.buffer.checkBounds: true [grpc-default-executor-0]
16:34:25.194 DEBUG [ResourceLeakDetectorFactory.java:195] - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@6a97c9f6 [grpc-default-executor-0]
16:34:25.313 DEBUG [DefaultChannelId.java:79] - -Dio.netty.processId: 68 (auto-detected) [grpc-default-executor-0]
16:34:25.319 DEBUG [NetUtil.java:139] - -Djava.net.preferIPv4Stack: false [grpc-default-executor-0]
16:34:25.319 DEBUG [NetUtil.java:140] - -Djava.net.preferIPv6Addresses: false [grpc-default-executor-0]
16:34:25.323 DEBUG [NetUtil.java:224] - Loopback interface: lo (lo, 127.0.0.1) [grpc-default-executor-0]
16:34:25.324 DEBUG [NetUtil.java:271] - /proc/sys/net/core/somaxconn: 128 [grpc-default-executor-0]
16:34:25.328 DEBUG [DefaultChannelId.java:101] - -Dio.netty.machineId: 02:42:ac:ff:fe:19:00:02 (auto-detected) [grpc-default-executor-0]
16:34:25.378 DEBUG [Recycler.java:96] - -Dio.netty.recycler.maxCapacityPerThread: 4096 [grpc-default-executor-0]
16:34:25.379 DEBUG [Recycler.java:97] - -Dio.netty.recycler.maxSharedCapacityFactor: 2 [grpc-default-executor-0]
16:34:25.379 DEBUG [Recycler.java:98] - -Dio.netty.recycler.linkCapacity: 16 [grpc-default-executor-0]
16:34:25.379 DEBUG [Recycler.java:99] - -Dio.netty.recycler.ratio: 8 [grpc-default-executor-0]
16:34:25.427 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192} [grpc-nio-worker-ELG-1-2]
16:34:25.453 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041 [grpc-nio-worker-ELG-1-2]
16:34:25.607 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192} [grpc-nio-worker-ELG-1-2]
16:34:25.612 DEBUG [AbstractInternalLogger.java:193] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-1-2]
16:34:25.620 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041 [grpc-nio-worker-ELG-1-2]
16:34:25.633 DEBUG [AbstractInternalLogger.java:193] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND SETTINGS: ack=true [grpc-nio-worker-ELG-1-2]
16:34:25.682 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:authority: prometheus-proxy:50440, :path: /ProxyService/connectAgent, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-2]
16:34:25.747 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND DATA: streamId=3 padding=0 endStream=true length=5 bytes=0000000000 [grpc-nio-worker-ELG-1-2]
16:34:25.869 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-2]
16:34:25.890 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND DATA: streamId=3 padding=0 endStream=false length=5 bytes=0000000000 [grpc-nio-worker-ELG-1-2]
16:34:25.890 DEBUG [AgentClientInterceptor.kt:57] - Assigned agentId to Agent{agentId=1, agentName=Unnamed-8f37b0e44c26, proxyHost=prometheus-proxy:50440, adminService=AdminService{port=8093, paths=[/ping, /version, /healthcheck, /threaddump, /debug]}, metricsService=Disabled} [Agent Unnamed-8f37b0e44c26]
16:34:25.902 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-1-2]
16:34:25.916 INFO  [AgentGrpcService.kt:144] - Connected to proxy at prometheus-proxy:50440 using plaintext [Agent Unnamed-8f37b0e44c26]
16:34:25.940 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:authority: prometheus-proxy:50440, :path: /ProxyService/registerAgent, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-2]
16:34:25.942 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND DATA: streamId=5 padding=0 endStream=true length=48 bytes=000000002b0a01311214556e6e616d65642d3866333762306534346332361a1070726f6d6574686575732d70726f7879 [grpc-nio-worker-ELG-1-2]
16:34:25.971 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-2]
16:34:25.980 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND DATA: streamId=5 padding=0 endStream=false length=30 bytes=000000001908011212496e76616c6964206167656e7449643a20311a0131 [grpc-nio-worker-ELG-1-2]
16:34:25.982 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-1-2]
16:34:25.993 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND HEADERS: streamId=7 headers=GrpcHttp2OutboundHeaders[:authority: prometheus-proxy:50440, :path: /ProxyService/registerPath, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-2]
16:34:25.994 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND DATA: streamId=7 padding=0 endStream=true length=51 bytes=000000002e0a013112297b7b20696e76656e746f72795f686f73746e616d65207d7d5f6361647669736f725f6d657472696373 [grpc-nio-worker-ELG-1-2]
16:34:26.006 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND HEADERS: streamId=7 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-2]
16:34:26.007 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND DATA: streamId=7 padding=0 endStream=false length=29 bytes=000000001808011212496e76616c6964206167656e7449643a20311801 [grpc-nio-worker-ELG-1-2]
16:34:26.008 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND HEADERS: streamId=7 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-1-2]
16:34:26.009 INFO  [AgentPathManager.kt:65] - Registered cadvisor:8080/metrics as /{{ inventory_hostname }}_cadvisor_metrics [Agent Unnamed-8f37b0e44c26]
16:34:26.012 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND HEADERS: streamId=9 headers=GrpcHttp2OutboundHeaders[:authority: prometheus-proxy:50440, :path: /ProxyService/registerPath, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.26.0, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-2]
16:34:26.013 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] OUTBOUND DATA: streamId=9 padding=0 endStream=true length=56 bytes=00000000330a0131122e7b7b20696e76656e746f72795f686f73746e616d65207d7d5f6e6f64655f6578706f727465725f6d657472696373 [grpc-nio-worker-ELG-1-2]
16:34:26.024 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND HEADERS: streamId=9 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, agent-id: 1, grpc-encoding: identity, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false [grpc-nio-worker-ELG-1-2]
16:34:26.025 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND DATA: streamId=9 padding=0 endStream=false length=31 bytes=000000001a08011212496e76616c6964206167656e7449643a203118022001 [grpc-nio-worker-ELG-1-2]
16:34:26.026 DEBUG [AbstractInternalLogger.java:216] - [id: 0x08144715, L:/172.25.0.2:37058 - R:prometheus-proxy/172.25.0.4:50440] INBOUND HEADERS: streamId=9 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true [grpc-nio-worker-ELG-1-2]
16:34:26.026 INFO  [AgentPathManager.kt:65] - Registered node_exporter:9100/metrics as /{{ inventory_hostname }}_node_exporter_metrics [Agent Unnamed-8f37b0e44c26]

Answer 28 · 2022-11-07T08:35:21.000Z

Hi

I have exactly the same issue with the last release v1.14.1. I create a minimalist example to reproduce the problem there (https://github.com/vincedom/prom-proxy-nginx/tree/main).

I also investigate it more deeply. The problem seems that when using nginx we go through io.prometheus.proxy.ProxyServerTransportFilter->transportReady and io.prometheus.proxy.ProxyServerTransportFilter->transportTerminated for each api call.
So when the agent calls connectAgent the agentContext is created at the begin of the call and remove at the end, so when it calls registerAgent there was an error since agentContext is missing.
I make my example "works" by removing the call to proxy.removeAgentContext in transportTerminated, but of course this is not a true fix since it's generated memory leaks, however it's demonstrated that this is the right problem.
I can't go further since I don't known Kotlin neither gRPC.

According to me, your project aims to use Prometheus through firewall or more generally in non easily accessible installations so it would make sense, to have a robust connection that can go through proxy and reverse proxy.

Thanks by advance, and thanks for your great job.
Regards

Vincent