opentracing-contrib/nginx-opentracing

does not capture traces when clients see "upstream prematurely closed connection"

Closed this issue · 4 comments

balusarakesh commented

Hi,

  • We have 2 nginx services (client and server).
  • The Client acts as a cache to our apps and the server serves some files from the file system
  • We have enabled datadog APM on the server nginx

Issue:

  • Sometimes the client nginx sees errors like 2024/03/06 18:42:16 [error] 175183#175183: *2502663847 upstream prematurely closed connection while reading upstream, client: CLIENT_IP, server: HOST_NAME, request: "GET ENDPOINT HTTP/1.1", upstream: "https://UPSTREAM_HOST:443/ENDPOINT", host: "HOST_NAME"
  • When the above error happens for ENDPOINT we don't see any errors on the server-nginx
  • In fact we don't even see the above ENDPOINT on the server-nginx APM for that timestamp 2024/03/06 18:42:16
  • As you can see from the screenshot for the above mentioned timestamp we are not seeing any error/request for that endpoint

datadog_apm

  • Thank you
miry commented

Just for clarification, do you see any problem with requests around those errors (for example 5xx)?

You should not expect any server side logs, because those errors means they are on client side only. For example it was some sort keep-alive and client writes in closed socket.
Server in other side closed this connection because of inactivity or it was restarted/crashed and socket were not closed properly.

In same time client should reconnect and repeate reqeust.

TLDR: I would ignore those errors or learn about origin of it.

balusarakesh commented

Just for clarification, do you see any problem with requests around those errors (for example 5xx)?

  • We do not see any issues with the requests around those errors
You should not expect any server side logs, because those errors means they are on client side only. For example it was some sort keep-alive and client writes in closed socket.
Server in other side closed this connection because of inactivity or it was restarted/crashed and socket were not closed properly.

How could this be a client side error when the error clearly says upstream prematurely closed connection? this is caused by server-nginx here and I think it is reasonable we see some trace error logs on the server-nginx which closed that connection prematurely (for whatever reason).

Thank you for your response

miry commented

@balusarakesh It is on TCP/IP level, while traces on Application Layer. It is related timeouts and configurations of how long server side keeps connections from client and how often client keeps the connections open. I would more look to nginx forums and check how to improve configurations.
Traces triggers on HTTP request, while this type of message on creation/manage sockets.

balusarakesh commented

thank you, that makes sense