504 FAILURE_ORIGIN_RESET_CONNECTION with high load
tussinha opened this issue · 2 comments
Hi, We are using Zuul 2.1.5 in our service when in we route different url to different downstream services/servers.
Now, on high load we are facing 504s with nfstatus as FAILURE_ORIGIN_RESET_CONNECTION. Below is the passport log.
The issue seems to be different from what is mentioned in #560 as I can confirm that the fix is already in the version being used.
It would be very helpful if someone could guide me on what could be the issue.
The downstream server is perfectly fine as requests to the same server through different route is working perfectly. Only the requests through our app is causing 504s.
Passport logs:
{"endOfBatch":true,"level":"INFO","logger":"org.apache.logging.slf4j.Log4jLogger","loggerName":"xxxx.gateway.zuul.passport.PassportLoggingHandler","msg":"State after complete. ProxyAttempted, current-server-conns = 750, current-http-reqs = 174, status = 504, nfstatus = FAILURE_ORIGIN_RESET_CONNECTION, UUID = 1066002d-5ba3-466a-ab04-33fc852d7498, req = uri=https://xxxxxxxx:443/rd/AvMVuwqjDv8S~QOBGrge~yL7NCEqOC75Mv9v~zj~PP_m.gif, method=get, clientip=x.x.x.x, passport = CurrentPassport {start_ms=1665029811559, [+0=IN_REQ_HEADERS_RECEIVED, +72977=FILTERS_INBOUND_START, +343721=FILTERS_INBOUND_END, +441856=ORIGIN_CONN_ACQUIRE_START, +443791=ORIGIN_CONN_ACQUIRE_END, +469729=OUT_REQ_HEADERS_SENDING, +645923=OUT_REQ_HEADERS_SENT, +656653=IN_REQ_LAST_CONTENT_RECEIVED, +692502=OUT_REQ_LAST_CONTENT_SENDING, +699235=OUT_REQ_LAST_CONTENT_SENT, +106847772=ORIGIN_CH_INACTIVE, +106991906=FILTERS_OUTBOUND_START, +107133904=FILTERS_OUTBOUND_END, +107162399=OUT_RESP_HEADERS_SENDING, +107184070=OUT_RESP_LAST_CONTENT_SENDING, +107277146=OUT_RESP_HEADERS_SENT, +107278579=OUT_RESP_LAST_CONTENT_SENT, +107538181=SERVER_CH_CLOSE, +107540515=SERVER_CH_CLOSE, +107599707=NOW]}","thread":"Salamander-ClientToZuulWorker-20","threadId":75,"threadPriority":5,"timestamp":"2022-10-06T04:16:51.667Z","ts":1665029811667}
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.
This issue was closed because it has been stalled for 7 days with no activity.