Internal write forwarding error on Aurora MySQL

[StanislavMakhrov]

Prerequisites:
Multiregional Aurora MySQL DB in AWS with 2 regions. For example - master region USW2 and slave region EUC1 (read only) with configure write forwarding.
Problem reproduces only in slave region where is required write forwarding to region in which are possible write operations, according above - USW2.
Every SQL transaction should have instruction:

"SET aurora_replica_read_consistency = 'EVENTUAL'"

1. Create simple project with 2 entities Parent and child
2. Deploy to AWS.
3. Save in DB parent entity with some child.
4. Save in DB orphaned child entity
5. In EUC1 region in the parent entity change current child to orphaned child entity

Expected:
Parent with new link to child was saved

Actual:

io.vertx.mysqlclient.MySQLException: Internal write forwarding error
io.vertx.mysqlclient.impl.codec.CommandCodec.decodeErrorPacketPayload(CommandCodec.java:134)
io.vertx.mysqlclient.impl.codec.CommandCodec.handleErrorPacketPayload(CommandCodec.java:123)
i.v.m.i.codec.ExtendedQueryCommandBaseCodec.handleInitPacket(ExtendedQueryCommandBaseCodec.java:42)
i.v.m.impl.codec.QueryCommandBaseCodec.decodePayload(QueryCommandBaseCodec.java:56)
i.v.m.impl.codec.ExtendedQueryCommandCodec.decodePayload(ExtendedQueryCommandCodec.java:108)
io.vertx.mysqlclient.impl.codec.MySQLDecoder.decodePackets(MySQLDecoder.java:72)\n\tat io.vertx.mysqlclient.impl.codec.MySQLDecoder.channelRead(MySQLDecoder.java:48)\n\tat i.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat i.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat i.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat i.n.c.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat i.n.c.n.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat i.n.u.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat i.n.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:829)\n\tat com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:705)\n\t... 1 frames truncated

[vietj]

can you reproduce this with docker ?

[StanislavMakhrov]

I'm afraid of there is no docker with multiregional AWS Aurora SQL emulation.

[StanislavMakhrov]

Adding here additional logs:

{"timestamp":"2023-07-08T21:08:09.573Z","level":"ERROR","message":"HTTP Request to /canary-api/canary failed, error id: 88a098a9-cf08-45a5-97b3-8eb9cac0e4a1-1","logger":"io.quarkus.vertx.http.runtime.QuarkusErrorHandler","thread":"vert.x-eventloop-thread-1","stack_trace":"io.smallrye.mutiny.CompositeException: Multiple exceptions caught:\n\t[Exception 0] javax.persistence.PersistenceException: org.hibernate.HibernateException: io.vertx.mysqlclient.MySQLException: Failed to send write forwarding request to Master\n\t[Exception 1] io.vertx.mysqlclient.MySQLException: Failed to send write forwarding request to Master\n\tat io.smallrye.mutiny.groups.UniOnFailure.lambda$call$3(UniOnFailure.java:108)\n\tat i.s.c.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)\n\tat i.s.m.o.u.UniOnFailureTransform$UniOnFailureTransformProcessor.onFailure(UniOnFailureTransform.java:54)\n\tat i.s.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)\n\tat i.s.m.o.u.b.UniCreateFromCompletionStage$CompletionStageUniSubscription.forwardResult(UniCreateFromCompletionStage.java:58)\n\tat java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)\n\tat j.u.c.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)\n\tat io.vertx.core.Future.lambda$toCompletionStage$3(Future.java:386)\n\tat io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153)\n\tat i.v.core.impl.future.FutureImpl$ListenerArray.onFailure(FutureImpl.java:268)\n\tat io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75)\n\tat io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230)\n\tat io.vertx.core.impl.future.FixedMapping.onFailure(FixedMapping.java:36)\n\tat io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75)\n\tat io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230)\n\tat io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23)\n\tat io.vertx.core.Promise.fail(Promise.java:89)\n\tat io.vertx.sqlclient.impl.TransactionImpl.lambda$txCommand$2(TransactionImpl.java:156)\n\tat io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153)\n\tat io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69)\n\tat i.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\n\tat i.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\n\tat i.n.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)\n\tat i.n.u.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat i.n.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:829)\n\t... 2 frames truncated\n\tSuppressed: io.vertx.mysqlclient.MySQLException: Failed to send write forwarding request to Master\n\t\tat io.vertx.mysqlclient.impl.codec.CommandCodec.decodeErrorPacketPayload(CommandCodec.java:134)\n\t\tat io.vertx.mysqlclient.impl.codec.CommandCodec.handleErrorPacketPayload(CommandCodec.java:123)\n\t\tat i.v.m.impl.codec.SimpleQueryCommandCodec.handleInitPacket(SimpleQueryCommandCodec.java:53)\n\t\tat i.v.m.impl.codec.QueryCommandBaseCodec.decodePayload(QueryCommandBaseCodec.java:56)\n\t\tat io.vertx.mysqlclient.impl.codec.MySQLDecoder.decodePackets(MySQLDecoder.java:72)\n\t\tat io.vertx.mysqlclient.impl.codec.MySQLDecoder.channelRead(MySQLDecoder.java:48)\n\t\tat i.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\t\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\t\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\t\tat i.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\t\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\t\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\t\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\t\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\t\tat i.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\t\tat i.n.c.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\t\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\t\tat i.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\t\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\t\tat i.n.c.n.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\t\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\t\tat io.netty.channel.nio.NioEventLoop...\n","level_value":40000}

The most interesting thing above: "Failed to send write forwarding request to Master"

[vietj]

thanks we can try to dig this error and see if it can reproduced with docker

[StanislavMakhrov]

@vietj I have updates. Revealed that it's not quarkus problem. Problem is on AWS side. Their service Aurora MySQL works incorrectly in multiregional setup. UPDATE sql statement of entity leads to restart of DB cluster.
The problem that we have 2 regions (USW2 as primary and EUC1 as slave) and in EUC1 region (read only) write-forwarding mechanism to USW2 (read/write) works incorrectly - leads to error and restart of DB cluster.

But I guess our problem could be solved by workaround - second reactive data source. But unfortunately according to official doc https://quarkus.io/guides/datasource#configure-multiple-datasources

Creating multiple reactive datasources is not currently possible.

Do we have a chance in some soon release to have such feature? We really need this. Thanks!

[tsegismont]

Creating multiple reactive datasources is not currently possible.

The feature has been implemented for some time now but this doc comment has not been removed.

See quarkusio/quarkus#34683

[tsegismont]

Closing as it seems this is not a Vert.x MySQL Client bug. Please reopen if I'm wrong.

[vietj]

thanks for keeping us up to date @StanislavMakhrov