When Webhook gets a disconnect, container doesnt restart or reinitialize?
Closed this issue · 6 comments
Overview: The webhook that connects to get this information will periodically drop sometimes due to network or ip hiccups. The container is still active and healthy, but does not know that the webhook is down and still remains in a healthy state
What it does: The server will sometimes drop the webhook due to outages but does not reestablish, or make the container dirty for it to auto restart.
What I want it to do: 1 of 2 things. EITHER the codebehind should re establish the connection OR the container should become not healthy and trigger a restart.
Reproduction:
- Establish server.
- kill someone for confirmation of km.
- turn on or off router or switch to replicate a network outage.
- notice that the container is still running
- Kill same reportable person
- Notice the notification does not appear.
- manually trigger restart
- kill person a 3rd time
- notice that the most recent km appears.
I have the same issue with webhooks. I run via docker-compose up -d and watch the logs with docker-compose logs -f, but eventually this command exits saying the process returned some code. Then, I re-run docker-compose logs -f and see the following stacktrace printed for each killmail received:
discordbot_1 | at ClientRequest.<anonymous> (/workspace/node_modules/ws/lib/websocket.js:888:7)
discordbot_1 | at ClientRequest.emit (node:events:513:28)
discordbot_1 | at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:693:27)
discordbot_1 | at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
discordbot_1 | at TLSSocket.socketOnData (node:_http_client:534:22)
discordbot_1 | at TLSSocket.emit (node:events:513:28)
discordbot_1 | at addChunk (node:internal/streams/readable:315:12)
discordbot_1 | at readableAddChunk (node:internal/streams/readable:289:9)
discordbot_1 | at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
discordbot_1 | at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
discordbot_1 | Emitted 'error' event on WebSocket instance at:
discordbot_1 | at emitErrorAndClose (/workspace/node_modules/ws/lib/websocket.js:1004:13)
discordbot_1 | at processTicksAndRejections (node:internal/process/task_queues:83:21)
discordbot_1 | error Command failed with exit code 1.
discordbot_1 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
discordbot_1 | yarn run v1.22.19
discordbot_1 | $ cd dist && node index
discordbot_1 | node:events:491
discordbot_1 | throw er; // Unhandled 'error' event```
I have the bot and made changes, but the zkillsubscriber still uses the same underlying webhook for connection to zkill's ws endpoint. The time between starting it (and it operating correctly) and starting to error-out is nondeterministic (sometimes it last a few hours, sometimes a day or longer).
On my fork, I added this.websocket.onerror handler to handle any error that occurs. I also made this.websocket.onclose perform a reconnect whenever the websocket gets disconnected.
Sorry for the late answer. I am currently pretty busy irl.
I'll try to look into this coming weekend.
@ocn feel free to make a pull request if you want to contribute your work