altoo-ag/akka-kryo-serialization

DataFormatException: incorrect header check

ekfliu opened this issue · 4 comments

if run with akka.actor.serialize-messages = on and post-serialization-transformations = "deflate" for single node serialization test results in error
java.util.zip.DataFormatException: incorrect header check at java.util.zip.Inflater.inflateBytes(Native Method) ~[na:1.8.0_112] at java.util.zip.Inflater.inflate(Inflater.java:259) ~[na:1.8.0_112] at java.util.zip.Inflater.inflate(Inflater.java:280) ~[na:1.8.0_112] at com.romix.akka.serialization.kryo.ZipKryoCompressor.fromBinary(KryoSerializer.scala:131) ~[akka-kryo-serialization_2.11-0.5.0.jar!/:na] at com.romix.akka.serialization.kryo.KryoTransformer$$anonfun$4$$anonfun$apply$2.apply(KryoSerializer.scala:56) ~[akka-kryo-serialization_2.11-0.5.0.jar!/:na] at com.romix.akka.serialization.kryo.KryoTransformer$$anonfun$4$$anonfun$apply$2.apply(KryoSerializer.scala:56) ~[akka-kryo-serialization_2.11-0.5.0.jar!/:na] at com.romix.akka.serialization.kryo.KryoTransformer.fromBinary(KryoSerializer.scala:63) ~[akka-kryo-serialization_2.11-0.5.0.jar!/:na] at com.romix.akka.serialization.kryo.KryoSerializer.fromBinary(KryoSerializer.scala:351) ~[akka-kryo-serialization_2.11-0.5.0.jar!/:na] at akka.serialization.Serialization$$anonfun$deserialize$3.apply(Serialization.scala:204) ~[akka-actor_2.11-2.5.4.jar!/:na] at scala.util.Try$.apply(Try.scala:192) ~[scala-library-2.11.11.jar!/:na] at akka.serialization.Serialization.deserialize(Serialization.scala:204) ~[akka-actor_2.11-2.5.4.jar!/:na] at akka.actor.dungeon.Dispatch$class.serializeAndDeserializePayload(Dispatch.scala:170) ~[akka-actor_2.11-2.5.4.jar!/:na] at akka.actor.dungeon.Dispatch$class.serializeAndDeserialize(Dispatch.scala:150) ~[akka-actor_2.11-2.5.4.jar!/:na] at akka.actor.dungeon.Dispatch$class.sendMessage(Dispatch.scala:133) ~[akka-actor_2.11-2.5.4.jar!/:na] at akka.actor.ActorCell.sendMessage(ActorCell.scala:370) ~[akka-actor_2.11-2.5.4.jar!/:na] at akka.actor.Cell$class.sendMessage(ActorCell.scala:291) ~[akka-actor_2.11-2.5.4.jar!/:na] at akka.actor.ActorCell.sendMessage(ActorCell.scala:370) ~[akka-actor_2.11-2.5.4.jar!/:na] at akka.actor.LocalActorRef.$bang(ActorRef.scala:400) ~[akka-actor_2.11-2.5.4.jar!/:na] at akka.actor.ActorRef.tell(ActorRef.scala:124) ~[akka-actor_2.11-2.5.4.jar!/:na]

This bug occurs even on non-same machine nodes in cluster, same machine, non-same actorSystem.

Could you provide a simple test which reproduces this issue?

I moved away from this dependency, wrote my own kryo-serialization support for akka.
But i was hit by same issue =)
The problem is that Deflator is not threadsafe, here

lazy val deflater = new Deflater(Deflater.BEST_SPEED)

you should not use it as field.
You should create new Deflator/Inflator on each serialization cycle.

This is the variant i wrote for my project:
`
private byte[] deflate( byte[] bytes ){
if( !compress ) return bytes;

	if( COMPRESSION_THRESHOLD > bytes.length )
	{
		return ByteString.createBuilder()
				.putByte( (byte)0 )
				.putBytes( bytes )
				.result()
				.toArray();
	}

	Deflater deflater = new Deflater( Deflater.BEST_SPEED );
	deflater.setInput( bytes );
	deflater.finish();
	ByteStringBuilder builder = ByteString.createBuilder()
			.putByte( (byte)1 );
	byte[] output = new byte[BLOCK_SIZE];
	while( !deflater.finished() )
	{
		int n = deflater.deflate( output );
		builder.putBytes( output, 0, n );
	}
	deflater.end();
	ByteString result = builder.result();
	log.trace( "Compressed from {} to {} bytes", bytes.length, result.size() );
	return result.toArray();
}

`
This is (probably) not optimal, but my servers are running for month 24/7 with no incident.

@lastrix thanks for sharing 👍