Traces insertion error
deathalt opened this issue · 4 comments
deathalt commented
Sometimes with no reason I can see this error
qryn-write-6478699f66-bqh54 {"level":30,"time":1718178232495,"pid":19,"hostname":"qryn-write-6478699f66-bqh54","name":"qryn","reqId":"req-1jvh","r
eq":{"method":"POST","url":"/v1/traces","hostname":"qryn-write.qryn.svc:3100","remoteAddress":"172.24.17.199","remotePort":51128},"msg":"incoming
request"}
qryn-write-6478699f66-7bbg9 {"level":50,"time":1718178232512,"pid":19,"hostname":"qryn-write-6478699f66-7bbg9","name":"qryn","reqId":"req-18is","e
rr":"Cannot read properties of undefined (reading 'forEach')\nTypeError: Cannot read properties of undefined (reading 'forEach')\n at flatVal (
/app/lib/utils.js:173:29)\n at flatAttrs (/app/lib/utils.js:187:7)\n at flatOTLPAttrs (/app/lib/utils.js:191:10)\n at new module.exports
(/app/lib/db/otlp.js:8:83)\n at /app/lib/db/clickhouse.js:362:32\n at Array.map (<anonymous>)\n at /app/lib/db/clickhouse.js:362:20\n
at new Promise (<anonymous>)\n at pushOTLP (/app/lib/db/clickhouse.js:358:10)\n at Object.handler (/app/lib/handlers/otlp_push.js:49:9)","ms
g":"Cannot read properties of undefined (reading 'forEach')"}
qryn-write-6478699f66-7bbg9 {"level":30,"time":1718178232512,"pid":19,"hostname":"qryn-write-6478699f66-7bbg9","name":"qryn","reqId":"req-18is","r
es":{"statusCode":500},"responseTime":691.9293155670166,"msg":"request completed"}
on clickhouse side I can see
<12>1 2024-06-12T07:42:30.866Z clickhouse-9 - 5842 HedgedConnectionsFactory - 2024.06.12 07:42:30.866114 [ 443770 ] {17D578E2466F2BB2} <Warning> HedgedConnectionsFactory: Connection failed at try №1, reason: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket (:43520 -> :9000). (NETWORK_ERROR) (version 23.9.2.56 (official build))
deathalt commented
same issue on 3.2.22 as well as on 3.2.20
akvlad commented
should be fixed in 3.2.23