sky-uk/kafka-message-scheduler

Uplift Kamon version from 1.1.6 to 2.1.1

peter-hazell opened this issue · 1 comments

Background

We have seen the following error when memory usage is high:

fi.iki.elonen.NanoHTTPD$Response send
SEVERE: Could not send response to the client
java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socket

According to previous investigations into the same error, this is because KMS Kamon is closing the connection with prometheus, resulting in errors and lost metrics.

We believe the above is due to the app using too much memory, effectively slowing it down causing the connection to prometheus to close.

kamon-io/Kamon#754 mentions the error and 2.1.1 of kamon removes NanoHTTP (kms is on 1.1.6)

Acceptance Criteria

  • Uplift Kamon version from 1.1.6 to 2.1.1 which removes NanoHTTP, which should stop errors when prometheus is scraping the app

Issue has been fixed, see #101 (comment) for notes