Fuzz test: failover_mito_regions is broken

Question

Fuzz test: failover_mito_regions is broken

evenyag opened this issue 11 days ago · 0 comments

evenyag commented 11 days ago

What type of bug is this?

Other

What subsystems are affected?

Other

Minimal reproduce step

Run the CI

What did you expect to see?

Fuzz Test with Chaos (Distributed, Remote WAL, fuzz_failover_mito_regions) should pass

What did you see instead?

https://github.com/GreptimeTeam/greptimedb/actions/runs/12373096778/job/34532913835#logs

What operating system did you use?

Unrelated

What version of GreptimeDB did you use?

latest

Relevant log output and stack trace

2024-12-17T12:48:03.835548872Z stdout F 1: Retry later
2024-12-17T12:48:03.835553921Z stdout F 2: Failed to request RegionServer my-greptimedb-datanode-1.my-greptimedb-datanode.my-greptimedb:4001, code: The service is currently unavailable, at src/client/src/region.rs:210:31
2024-12-17T12:48:03.83555865Z stdout F 3: error trying to connect: dns error: failed to lookup address information: Name or service not known, at src/client/src/error.rs:171:23
2024-12-17T12:48:03.835738884Z stdout F 2024-12-17T12:48:03.835590Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-7f3fa1ee-c103-4499-8353-ec104d16a902 retry for the 5 times after 9787 millis
2024-12-17T12:48:04.007623553Z stdout F 2024-12-17T12:48:04.007439Z ERROR LocalManager::submit_root_procedure: common_procedure::local::runner: Failed to execute procedure metasrv-procedure::CreateTable-83a5a9d5-ad05-4d87-8380-07bd64109ce0, retry: true err=0: Procedure exec failed
2024-12-17T12:48:04.007645805Z stdout F 1: Retry later
2024-12-17T12:48:04.007650985Z stdout F 2: Failed to request RegionServer my-greptimedb-datanode-1.my-greptimedb-datanode.my-greptimedb:4001, code: The service is currently unavailable, at src/client/src/region.rs:210:31
2024-12-17T12:48:04.007655844Z stdout F 3: error trying to connect: dns error: failed to lookup address information: Name or service not known, at src/client/src/error.rs:171:23
2024-12-17T12:48:04.00780869Z stdout F 2024-12-17T12:48:04.007691Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-83a5a9d5-ad05-4d87-8380-07bd64109ce0 retry for the 5 times after 8990 millis
2024-12-17T12:48:04.466782221Z stdout F 2024-12-17T12:48:04.466526Z ERROR LocalManager::submit_root_procedure: common_procedure::local::runner: Failed to execute procedure metasrv-procedure::CreateTable-bfaac72c-303a-438c-9a66-ec0d733999a1, retry: true err=0: Procedure exec failed
2024-12-17T12:48:04.466796989Z stdout F 1: Retry later
2024-12-17T12:48:04.466801107Z stdout F 2: Failed to request RegionServer my-greptimedb-datanode-1.my-greptimedb-datanode.my-greptimedb:4001, code: The service is currently unavailable, at src/client/src/region.rs:210:31
2024-12-17T12:48:04.466804734Z stdout F 3: error trying to connect: dns error: failed to lookup address information: Name or service not known, at src/client/src/error.rs:171:23
2024-12-17T12:48:04.467032231Z stdout F 2024-12-17T12:48:04.466914Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-bfaac72c-303a-438c-9a66-ec0d733999a1 retry for the 5 times after 8725 millis
2024-12-17T12:48:10.779311794Z stdout F 2024-12-17T12:48:10.779134Z  WARN meta_srv::region::supervisor: Detects region failures: [(0, 0, 4672924418048(1088, 0)), (0, 0, 4642859646976(1081, 0)), (0, 0, 4647154614272(1082, 0)), (0, 0, 4655744548864(1084, 0)), (0, 0, 4660039516160(1085, 0)), (0, 0, 4664334483456(1086, 0)), (0, 0, 4668629450752(1087, 0)), (0, 0, 4651449581568(1083, 0))]
2024-12-17T12:48:10.779628261Z stdout F 2024-12-17T12:48:10.779520Z ERROR meta_srv::region::supervisor: Failed to execute region failover for region: 4672924418048(1088, 0), datanode: 0 err=0: Invalid arguments: The `from_peer_id` can't equal `to_peer_id`, at src/meta-srv/src/procedure/region_migration/manager.rs:188:14
2024-12-17T12:48:10.780904817Z stdout F 2024-12-17T12:48:10.780797Z ERROR meta_srv::region::supervisor: Failed to execute region failover for region: 4647154614272(1082, 0), datanode: 0 err=0: Invalid arguments: The `from_peer_id` can't equal `to_peer_id`, at src/meta-srv/src/procedure/region_migration/manager.rs:188:14
2024-12-17T12:48:10.782142568Z stdout F 2024-12-17T12:48:10.782018Z ERROR meta_srv::region::supervisor: Failed to execute region failover for region: 4660039516160(1085, 0), datanode: 0 err=0: Invalid arguments: The `from_peer_id` can't equal `to_peer_id`, at src/meta-srv/src/procedure/region_migration/manager.rs:188:14
2024-12-17T12:48:10.783446135Z stdout F 2024-12-17T12:48:10.783333Z ERROR meta_srv::region::supervisor: Failed to execute region failover for region: 4668629450752(1087, 0), datanode: 0 err=0: Invalid arguments: The `from_peer_id` can't equal `to_peer_id`, at src/meta-srv/src/procedure/region_migration/manager.rs:188:14
2024-12-17T12:48:11.780501426Z stdout F 2024-12-17T12:48:11.779541Z  WARN meta_srv::region::supervisor: Detects region failures: [(0, 0, 4672924418048(1088, 0)), (0, 0, 4647154614272(1082, 0)), (0, 0, 4660039516160(1085, 0)), (0, 0, 4668629450752(1087, 0))]
2024-12-17T12:48:11.780537463Z stdout F 2024-12-17T12:48:11.779913Z ERROR meta_srv::region::supervisor: Failed to execute region failover for region: 4672924418048(1088, 0), datanode: 0 err=0: Invalid arguments: The `from_peer_id` can't equal `to_peer_id`, at src/meta-srv/src/procedure/region_migration/manager.rs:188:14
2024-12-17T12:48:11.781770826Z stdout F 2024-12-17T12:48:11.781645Z ERROR meta_srv::region::supervisor: Failed to execute region failover for region: 4660039516160(1085, 0), datanode: 0 err=0: Invalid arguments: The `from_peer_id` can't equal `to_peer_id`, at src/meta-srv/src/procedure/region_migration/manager.rs:188:14
2024-12-17T12:48:12.77943368Z stdout F 2024-12-17T12:48:12.779272Z  WARN meta_srv::region::supervisor: Detects region failures: [(0, 0, 4672924418048(1088, 0)), (0, 0, 4660039516160(1085, 0))]
2024-12-17T12:48:12.779692636Z stdout F 2024-12-17T12:48:12.779576Z ERROR meta_srv::region::supervisor: Failed to execute region failover for region: 4672924418048(1088, 0), datanode: 0 err=0: Invalid arguments: The `from_peer_id` can't equal `to_peer_id`, at src/meta-srv/src/procedure/region_migration/manager.rs:188:14
2024-12-17T12:48:13.019724308Z stdout F 2024-12-17T12:48:13.019520Z ERROR LocalManager::submit_root_procedure: common_procedure::local::runner: Failed to execute procedure metasrv-procedure::CreateTable-83a5a9d5-ad05-4d87-8380-07bd64109ce0, retry: true err=0: Procedure exec failed
2024-12-17T12:48:13.01974171Z stdout F 1: Retry later
2024-12-17T12:48:13.019747251Z stdout F 2: Failed to request RegionServer my-greptimedb-datanode-1.my-greptimedb-datanode.my-greptimedb:4001, code: The service is currently unavailable, at src/client/src/region.rs:210:31
2024-12-17T12:48:13.01975245Z stdout F 3: error trying to connect: dns error: failed to lookup address information: Name or service not known, at src/client/src/error.rs:171:23
2024-12-17T12:48:13.019973176Z stdout F 2024-12-17T12:48:13.019859Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-83a5a9d5-ad05-4d87-8380-07bd64109ce0 retry for the 6 times after 16103 millis
2024-12-17T12:48:13.127377727Z stdout F 2024-12-17T12:48:13.127162Z ERROR LocalManager::submit_root_procedure: common_procedure::local::runner: Failed to execute procedure metasrv-procedure::CreateTable-de0da9b8-9861-45e4-b830-4588ae88472f, retry: true err=0: Procedure exec failed
2024-12-17T12:48:13.127396292Z stdout F 1: Retry later
2024-12-17T12:48:13.127401862Z stdout F 2: Failed to request RegionServer my-greptimedb-datanode-1.my-greptimedb-datanode.my-greptimedb:4001, code: The service is currently unavailable, at src/client/src/region.rs:210:31
2024-12-17T12:48:13.127406541Z stdout F 3: error trying to connect: dns error: failed to lookup address information: Name or service not known, at src/client/src/error.rs:171:23
2024-12-17T12:48:13.127601675Z stdout F 2024-12-17T12:48:13.127500Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-de0da9b8-9861-45e4-b830-4588ae88472f retry for the 6 times after 16245 millis
2024-12-17T12:48:13.195986519Z stdout F 2024-12-17T12:48:13.195763Z ERROR LocalManager::submit_root_procedure: common_procedure::local::runner: Failed to execute procedure metasrv-procedure::CreateTable-bfaac72c-303a-438c-9a66-ec0d733999a1, retry: true err=0: Procedure exec failed
2024-12-17T12:48:13.19600271Z stdout F 1: Retry later
2024-12-17T12:48:13.19600849Z stdout F 2: Failed to request RegionServer my-greptimedb-datanode-1.my-greptimedb-datanode.my-greptimedb:4001, code: The service is currently unavailable, at src/client/src/region.rs:210:31
2024-12-17T12:48:13.19601365Z stdout F 3: error trying to connect: dns error: failed to lookup address information: Name or service not known, at src/client/src/error.rs:171:23
2024-12-17T12:48:13.196254795Z stdout F 2024-12-17T12:48:13.196137Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-bfaac72c-303a-438c-9a66-ec0d733999a1 retry for the 6 times after 16767 millis
2024-12-17T12:48:13.626355577Z stdout F 2024-12-17T12:48:13.626144Z ERROR LocalManager::submit_root_procedure: common_procedure::local::runner: Failed to execute procedure metasrv-procedure::CreateTable-7f3fa1ee-c103-4499-8353-ec104d16a902, retry: true err=0: Procedure exec failed
2024-12-17T12:48:13.626381235Z stdout F 1: Retry later
2024-12-17T12:48:13.626387667Z stdout F 2: Failed to request RegionServer my-greptimedb-datanode-1.my-greptimedb-datanode.my-greptimedb:4001, code: The service is currently unavailable, at src/client/src/region.rs:210:31
2024-12-17T12:48:13.626392727Z stdout F 3: error trying to connect: dns error: failed to lookup address information: Name or service not known, at src/client/src/error.rs:171:23
2024-12-17T12:48:13.626625812Z stdout F 2024-12-17T12:48:13.626524Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-7f3fa1ee-c103-4499-8353-ec104d16a902 retry for the 6 times after 18353 millis
2024-12-17T12:48:13.778922054Z stdout F 2024-12-17T12:48:13.778707Z  WARN meta_srv::region::supervisor: Detects region failures: [(0, 0, 4672924418048(1088, 0))]
2024-12-17T12:48:13.77921744Z stdout F 2024-12-17T12:48:13.779072Z ERROR meta_srv::region::supervisor: Failed to execute region failover for region: 4672924418048(1088, 0), datanode: 0 err=0: Invalid arguments: The `from_peer_id` can't equal `to_peer_id`, at src/meta-srv/src/procedure/region_migration/manager.rs:188:14
2024-12-17T12:48:14.779662855Z stdout F 2024-12-17T12:48:14.779505Z  WARN meta_srv::region::supervisor: Detects region failures: [(0, 0, 4672924418048(1088, 0))]
2024-12-17T12:48:29.149168369Z stdout F 2024-12-17T12:48:29.149017Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-83a5a9d5-ad05-4d87-8380-07bd64109ce0 retry for the 7 times after 37472 millis
2024-12-17T12:48:29.387376445Z stdout F 2024-12-17T12:48:29.387240Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-de0da9b8-9861-45e4-b830-4588ae88472f retry for the 7 times after 32456 millis
2024-12-17T12:48:29.979132046Z stdout F 2024-12-17T12:48:29.978984Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-bfaac72c-303a-438c-9a66-ec0d733999a1 retry for the 7 times after 35358 millis
2024-12-17T12:48:31.993956988Z stdout F 2024-12-17T12:48:31.993790Z  INFO LocalManager::submit_root_procedure: common_procedure::local::runner: Procedure metasrv-procedure::CreateTable-7f3fa1ee-c103-4499-8353-ec104d16a902 retry for the 7 times after 39054 millis