Akka 2.6.4 cluster fails to connect if Kamon is inited

[dparamoshkin]

20:26:24.199Z [ClusterSystem-akka.actor.default-dispatcher-38] WARN akka.remote.artery.Deserializer [] - Failed to deserialize message from [unknown] with serializer id [17] and manifest [d]. java.lang.IllegalArgumentException

[ranglang]

same error

[danischroeter]

We also get the same error and so I found this issue...
@dparamoshkin how did you identify this is caused by kamon? Did you already find out more about the issue?

We use akka 2.6.10 and kamon 2.1.8.

This seems to be a HandshakeReq see: akka.remote.serialization.ArteryMessageSerializer
The inner cause shows its thrown from java.nio.Buffer

private[akka] object ArteryMessageSerializer {
 ...
  private val HandshakeReqManifest = "d"

The contained errors look like these: (values differ)

java.lang.IllegalArgumentException: newPosition > limit: (1761607940 > 264)
java.lang.IllegalArgumentException: newPosition < 0: (-1241513799 < 0)

java.nio.Buffer

    /**
     * Verify that {@code 0 < newPosition <= limit}
     *
     * @param newPosition
     *        The new position value
     *
     * @throws IllegalArgumentException
     *         If the specified position is out of bounds.
     */
    private IllegalArgumentException createPositionException(int newPosition) {
        String msg = null;

        if (newPosition > limit) {
            msg = "newPosition > limit: (" + newPosition + " > " + limit + ")";
        } else { // assume negative
            assert newPosition < 0 : "newPosition expected to be negative";
            msg = "newPosition < 0: (" + newPosition + " < 0)";
        }

        return new IllegalArgumentException(msg);
    }

[danischroeter]

@ivantopo most probably a bug in SerializeForArteryAdvice. Wdyt?
The weird thing is, that even though we see this in the logs we did not have any bad effects so far... 🤔

[dparamoshkin]

We also get the same error and so I found this issue...
@dparamoshkin how did you identify this is caused by kamon? Did you already find out more about the issue?

For me this issue happened after trying to introduce kamon metrics to our application, problem is we one node is started as tomcat application and other nodes as normal java application (with main method)

[SimunKaracic]

We're working on fixing this, it's almost done! :)

[SimunKaracic]

Fixed via #917, released in 2.1.10

[fggarcia]

Hi @SimunKaracic today i release my app with kamon 2.1.11. I try to upgrade from kamon 2.19 (all my production instances have this version).

When the first deploy node is updated to the new version i have the same issue that @dparamoshkin

2021-02-01T21:05:30.135Z (xxxx) WARN  [][][xxx-akka.actor.default-dispatcher-58] akka.remote.artery.Deserializer - Failed to deserialize message from [unknown] with serializer id [17] and manifest [d]. java.lang.IllegalArgumentException: newPosition < 0: (-548456067 < 0)

does it exist any way to deploy without a full cluster restart??

Prev release dependencies:

val akkaVersion = "2.6.10"
val akkaHttpVersion = "10.2.2"
val kamonVersion = "2.1.9"

new release dependencies:

val akkaVersion = "2.6.12"
val akkaHttpVersion = "10.2.3"
val kamonVersion = "2.1.11"