xline-kv/Xline

[Bug]: CI nextest failed by timeout

Opened this issue · 3 comments

lxl66566 commented

Description about the bug

There was a case that nextest failed by timeout found on github workflow. When I reruned the code without modifing any code, the failure disappeared.
It's a rare error that cannot be reproduced stably. And It can be either the code bug or a nextest bug.

Version

0.6.1 (Default)

Relevant log output

the failed CI: https://github.com/xline-kv/Xline/actions/runs/10003614333/job/27650819920?pr=905
the success(rerun) CI: https://github.com/xline-kv/Xline/actions/runs/10003614333?pr=905
The failed test is: curp::it server::shutdown_rpc_should_shutdown_the_cluster
There're a lot of RpcTransport(()) error

Code of Conduct

  • I agree to follow this project's Code of Conduct
github-actions commented

👋 Thanks for opening this issue!

Reply with the following command on its own line to get help or engage:

  • /contributing-agreement : to print Contributing Agreements.
  • /assignme : to assign this issue to you.
lxl66566 commented

That might caused by the following:

  • A put client opens a thread and sends 10 Put requests one by one.
  • Another client proposed shutdown to cluster
  • The putclient waits for a response, exceeding the time limit. now the cluster may have shut down successfully, and RPC closed.
  • The put client retries, and gets an RpcTransport error, keep retrying. Each put request may retry 3 times, 10.5s.
  • each 10 put request retry for at most 105s, exceed 30s test timeout.

This should be fixed when #918 merge.

github-actions commented

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 14 days.