Ensure all servers listen on 0.0.0.0

[yurishkuro]

The upstream issue with 0.0.0.0 vs. local host strikes again (#6207).

• ensure that all servers are listening on 0.0.0.0 (or :port, which is equivalent)
• ensure that all servers log this endpoint at startup (we may be only logging ports in some places)
• why didn't our CI for hotrod/all-in-one catch this?

Related open-telemetry/opentelemetry-collector#11713

[pradeeptac]

Hi @yurishkuro ,
Few observations,

1. the tests for all-in-one does not include any tests for OTLP exporter. Hence the CI did not catch it.
2. Without the explicit over-ride for receivers.otlp.protocols.http.endpoint and receivers.otlp.protocols.grpc.endpoint to 0.0.0.0:4318 and 0.0.0.0:4317, the jaeger:2.0.0 (all-in-one v2) image starts the OTLP collectors at localhost:4318 and localhost:4317 for http and grpc respectively which can be determined by the logs of docker image startup
   2024-11-19T17:31:24.412Z info otlpreceiver@v0.112.0/otlp.go:112 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "localhost:4317"} 2024-11-19T17:31:24.412Z info otlpreceiver@v0.112.0/otlp.go:169 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "localhost:4318"} 2024-11-19T17:31:24.413Z info service@v0.112.0/service.go:230 Everything is ready. Begin running and processing data.
3. The same can also be confirmed by running a curl command

• when jaeger is started with --set receivers.otlp.protocols.http.endpoint=0.0.0.0:4318 --set receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317 over-rides
  jaeger % curl -X GET http://0.0.0.0:4318/ 404 page not found jaeger %
• without the --set receivers.otlp.protocols.http.endpoint=0.0.0.0:4318 --set receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317 over-rides
  jaeger % curl -X GET http://localhost:4318/ curl: (56) Recv failure: Connection reset by peer jaeger % curl -X GET http://0.0.0.0:4318/ curl: (56) Recv failure: Connection reset by peer jaeger %

4. The OTLP receiver does indeed log at which hostPort it is starting. The OTLP receiver gets initialised at collector by invoking the opentelemetry otlp Start method . This starts the GRPC and HTTP servers which logs the hostPort at which otlp reciever is starting.
   2024-11-19T18:33:27.386Z info otlpreceiver@v0.112.0/otlp.go:169 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "localhost:4318"}
5. On Jaeger default flags, we only specify :4318 and :4317 for hostPort for otlp collector and according to OTLP Receiver documentation the default the default hostPort is localhost:4317 for grpc protocol, localhost:4318 http protocol .

Possible Enhancements:

1. Include otlp collector integration test in https://github.com/jaegertracing/jaeger/blob/main/cmd/all-in-one/all_in_one_test.go
2. Specify hostPort explicitly to be 0.0.0.0:4318 and 0.0.0.0:4317 for HTTP and GRPC flags respectively https://github.com/jaegertracing/jaeger/blob/main/cmd/collector/app/flags/flags.go#L186

Let me know your thoughts on above. Thanks!!

PS: new to Jaeger, thought would give this a try to get started.

[yurishkuro]

1. I was referring to HotROD integration tests that includes hotrod app sending data via OTLP to all-in-one
2. I am aware of those two ports, my question was about all other ports that jaeger opens
3. your curl command is not authoritative as some of the endpoints do not accept GET requests so getting 404 does not mean misconfiguration. If anything it confirms the opposite - the server has to be reachable to respond with 404.
4. same point as in 2 above

[pradeeptac]

[1] I was referring to HotROD integration tests that includes hotrod app sending data via OTLP to all-in-one ---> Thanks for clarifying! will check the HotROD Integration test and update the conversation.

[3] the server has to be reachable to respond with 404. ---> Yaa actually, my point was the same. 404 means reachable. and the curl gave 404 with the --set receivers.otlp.protocols.http.endpoint=0.0.0.0:4318 --set receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317 over-rides . Without setting above over-rides, it was giving "Recv failure: Connection reset by peer jaeger".

[2] and [4] question was about all other ports that jaeger opens ---> Understood. would check for the rest of the ports that Jaeger opens and if the hostPorts are logged.

appreciate your inputs! Thanks!

[pradeeptac]

Did a quick check on [1] , for the jaeger v2 docker CI, we do set the below over-rides in docker-compose-v2.yml . This should allow the hotrod example to send metrics to otlp collector over http://jaeger:4318.

- --set=receivers.otlp.protocols.grpc.endpoint="0.0.0.0:4317"
- --set=receivers.otlp.protocols.http.endpoint="0.0.0.0:4318"

[yurishkuro]

Right, but we added them after the OTEL upgrade was done and presumably our build was still green. I am verifying that now: #6226

[pradeeptac]

Thanks for taking it forward @yurishkuro ! Was trying to follow various issues around this.

1. #6231 ---> We are introducing an environment variable JAEGER_LISTEN_HOST which gets used from all-in-one/v2 config. This defaults to localhost when not defined. In case of running from docker, it overrides to 0.0.0.0
2. Suppress warning about 0.0.0.0 ---> As a result of running at port 0.0.0.0 from inside docker, we get an warning as below currently which needs to be suppressed appropriately.

Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.

3. ensure that all servers log this endpoint at startup (we may be only logging ports in some places) ---> Initial Point 3 of this Issue still needs to be answered. Will check and update the conversation.