netty/netty

Apparent corruption with gRPC and Netty 4.1.111

ejona86 opened this issue · 9 comments

ejona86 commented

Edit: This issue should be limited to grpc-java.

Still very early, and I can totally believe gRPC's behavior could be contributing, but getting the word out as we figure out what is wrong. The gRPC version doesn't seem to matter much.

I'm seeing the HTTP/2 frame inside the gRPC message. grpc/grpc-java#11284 (comment)

I've not yet reproduced it, but people are reporting there wasn't an issue with Netty 4.1.110. My current guess is the ByteBuf changes (fixes) in 4.1.111 may be causing gRPC's NettyAdaptiveCumulator to improperly reuse buffers or use the wrong indexes.

normanmaurer commented

@ejona86 keep me posted

franz1981 commented

We noticed something on quarkus as well, cc @cescoffier

lhotari commented

Root cause explained in this comment grpc/grpc-java#11284 (comment) by @sergiitk.

I suspect NettyAdaptiveCumulator got broken by #14093.
Because of #12844, we had a temporary workaround to correct r/w indexes, which never got resolved

normanmaurer commented

@franz1981 @cescoffier any more details ?

franz1981 commented

I don't have much more details actually: quarkusio/quarkus#41155 (comment)

normanmaurer commented

ah ok... its also because of grpc-java and not some issue with using netty directly.

cescoffier commented

@normanmaurer Yes, the Quarkus issue is related to grpc-java. Our "pure" gRPC server works nicely with 4.1.111.

ejona86 commented

This issue is limited to grpc-java.

I'm going to leave this open until patch releases of grpc-java are available, to catch attention of anybody that stumbles on the issue.

  • 👍1
ejona86 commented

grpc-java v1.65.0 is available, which has a fix for the corruption. Some other releases are still in-progress with the fix. But I think this issue has served its purpose, so closing.

  • 🎉1
  • 👀1