futurewei-cloud/chogori-platform

HTTP proxy assertion fail on shutdown with ongoing txns

Opened this issue · 0 comments

It seems the heartbeat for a txn is not cancelled when the txn is dropped in the HTTPProxy, causing the hb timer to attempt to use the RPCDispatcher to send a message. The dispatcher has shutdown at this point and asserts that it should not be used, e.g.

[0004:14:18:51.953.116]-http_proxy-[0]-(k2::skv_client) [DEBUG] [/build/src/k2/module/k23si/client/k23si_client.cpp:66 @operator()] send hb for mtr={timestamp={endCount=397131397989158, tsoId=100000, startDelta=100000}, priority=medium}
[0004:14:18:51.953.201]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:88 @partitionRequest] making partition request with deadline=0000:00:00:00.499.993
[0004:14:18:52.009.280]-http_proxy-[0]-(k2::transport) [ERROR] [/build/src/k2/transport/RPCDispatcher.h:189 @operator()] RPC send failed with uncaught exception: dispatcher has shut down
[0004:14:18:52.009.353]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:360 @operator()] partition call completed with status={code=500, message=unknown exception while sending request}
[0004:14:18:52.009.405]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:105 @operator()] call failed with retryable status: {code=503, message=unable to make call at this time}
[0004:14:18:52.009.423]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:110 @operator()] refreshing collection from CPO after status={code=503, message=unable to make call at this time}
[0004:14:18:52.010.636]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:247 @_getAssignedPartitionWithRetry] time remaining=0000:00:00:00.442.563, for coll=HTTPClient, key={schemaName=test_schema, partitionKey=^00^00^01, rangeKey=^00^00^01}, reverse=false, excludedKey=false, retries=1
[0004:14:18:52.013.215]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:273 @_getAssignedPartitionWithRetry] no existing waiter for name=HTTPClient. Creating new one
http_proxy: /usr/local/include/seastar/core/sharded.hh:755: Service& seastar::sharded::local() [with Service = k2::RPCDispatcher]: Assertion `local_is_initialized()' failed.