restart with `export DX_CLEAN=false` may fail
Opened this issue · 1 comments
HAOYUatHZ commented
description
restart with export DX_CLEAN=false may fail for
- prover-cluster client
- block_submitter
- tick
- restapi
- contracts ticker
how to reproduce
export DX_CLEAN=false
./stopsh
./run.sh
log
ubuntu@ip-172-31-18-140:~/repos/fluidex-backend$ make tail_log
docker-compose --file orchestra/docker/docker-compose.yaml --project-name orchestra logs > orchestra/docker-compose.log
docker-compose --file regnbue-bridge/docker/docker-compose.yaml --project-name faucet logs > regnbue-bridge/docker-compose.log
ls rollup-state-manager/*.log prover-cluster/*.log dingir-exchange/*.log dingir-exchange/logs/*.log regnbue-bridge/*.log contracts/*.log orchestra/*.log | xargs tail -n 3
==> contracts/ganache.2021-09-23.log <==
9007199254740991
Listening on 127.0.0.1:8545
==> contracts/ticker.2021-09-23.log <==
at emitErrorNT (node:internal/streams/destroy:193:8)
at emitErrorCloseNT (node:internal/streams/destroy:158:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
==> dingir-exchange/logs/matchengine.2021-09-23.log <==
Sep 23 04:10:38.953 INFO dingir_exchange::message::producer: kafka producer disconnected
Sep 23 04:10:38.953 INFO dingir_exchange::message::producer: kafka producer running terminated
==> dingir-exchange/logs/persistor.2021-09-23.log <==
Sep 23 04:10:40.281 INFO dingir_exchange::message::persist: start auto commiting for topic internaltransfer
Sep 23 04:10:40.281 INFO dingir_exchange::message::persist: start auto commiting for topic registeruser
Sep 23 04:10:41.301 INFO dingir_exchange::message::consumer: start consuming topic ["trades", "internaltransfer", "registeruser", "balances", "orders"]
==> dingir-exchange/logs/restapi.2021-09-23.log <==
44: __libc_start_main
45: _start
==> dingir-exchange/tick.2021-09-23.log <==
details: 'No connection established',
metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}
==> orchestra/docker-compose.log <==
exchange_envoy | [2021-09-23 04:19:48.260][19][debug][connection] [source/common/network/connection_impl.cc:243] [C49] closing socket: 0
exchange_envoy | [2021-09-23 04:19:48.260][19][debug][client] [source/common/http/codec_client.cc:107] [C49] disconnect. resetting 0 pending requests
exchange_envoy | [2021-09-23 04:19:48.260][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:407] [C49] client disconnected, failure reason:
==> prover-cluster/client.2021-09-23.log <==
15: __libc_start_main
16: _start
==> prover-cluster/coordinator.2021-09-23.log <==
status = $1
and updated_time < current_timestamp - interval '172800 seconds'
==> regnbue-bridge/block_submitter.2021-09-23.log <==
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111)
==> regnbue-bridge/docker-compose.log <==
regnbue_bridge_pq | 2021-09-23 04:10:26.580 UTC [28] LOG: TimescaleDB background worker launcher connected to shared catalogs
regnbue_bridge_pq | 2021-09-23 04:11:31.098 UTC [51] LOG: the "timescaledb" extension is not up-to-date
regnbue_bridge_pq | 2021-09-23 04:11:31.098 UTC [51] HINT: The most up-to-date version is 2.4.2, the installed version is 2.1.0.
==> regnbue-bridge/faucet.2021-09-23.log <==
Sep 23 04:19:50.229 INFO sqlx::query: COMMIT; rows: 0, elapsed: 93.348µs
Sep 23 04:19:50.230 INFO sqlx::query: /* SQLx ping */; rows: 0, elapsed: 83.309µs
==> rollup-state-manager/rollup_state_manager.2021-09-23.log <==
Sep 23 04:15:22.823 INFO rollup_state_manager: generate 0 blocks with block_size 2 in 240.00372s: average TPS: 0
Sep 23 04:17:22.823 INFO rollup_state_manager: generate 0 blocks with block_size 2 in 360.00385s: average TPS: 0
Sep 23 04:19:22.823 INFO rollup_state_manager: generate 0 blocks with block_size 2 in 480.00394s: average TPS: 0
ubuntu@ip-172-31-18-140:~/repos/fluidex-backend$ tail -n 300 dingir-exchange/logs/restapi.2021-09-23.log
Sep 23 04:10:40.256 DEBUG restapi: Prepared DB connection: postgres://exchange:exchange_AA9944@127.0.0.1/exchange
Sep 23 04:10:40.257 INFO restapi: Connect to manage channel http://0.0.0.0:50051
Sep 23 04:10:41.666 ERROR fluidex_common::non_blocking_tracing: thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))', src/bin/restapi.rs:34:18
0: fluidex_common::non_blocking_tracing::get_backtrace
at /home/ubuntu/.cargo/git/checkouts/common-rs-f10c3d305ff6aa0a/24293b3/src/non_blocking_tracing.rs:50:30
1: fluidex_common::non_blocking_tracing::set_panic_hook::{{closure}}
at /home/ubuntu/.cargo/git/checkouts/common-rs-f10c3d305ff6aa0a/24293b3/src/non_blocking_tracing.rs:39:13
2: std::panicking::rust_panic_with_hook
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:595:17
3: std::panicking::begin_panic_handler::{{closure}}
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:497:13
4: std::sys_common::backtrace::__rust_end_short_backtrace
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/sys_common/backtrace.rs:141:18
5: rust_begin_unwind
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:493:5
6: core::panicking::panic_fmt
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/panicking.rs:92:14
7: core::result::unwrap_failed
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/result.rs:1355:5
8: core::result::Result<T,E>::unwrap
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/result.rs:1037:23
9: restapi::main::{{closure}}
at src/bin/restapi.rs:29:13
10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/mod.rs:80:19
11: <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll::{{closure}}::{{closure}}
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:668:65
12: tokio::coop::with_budget::{{closure}}
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:106:9
13: std::thread::local::LocalKey<T>::try_with
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:376:16
14: std::thread::local::LocalKey<T>::with
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:352:9
15: tokio::coop::with_budget
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:99:5
tokio::coop::budget
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:76:5
<tokio::task::local::RunUntil<T> as core::future::future::Future>::poll::{{closure}}
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:668:42
16: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/macros/scoped_tls.rs:61:9
17: tokio::task::local::LocalSet::with
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:573:9
18: <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:658:9
19: tokio::task::local::LocalSet::run_until::{{closure}}
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:516:9
20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/mod.rs:80:19
21: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/future.rs:120:9
22: tokio::runtime::basic_scheduler::Inner<P>::block_on::{{closure}}::{{closure}}
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:208:62
23: tokio::coop::with_budget::{{closure}}
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:106:9
24: std::thread::local::LocalKey<T>::try_with
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:376:16
25: std::thread::local::LocalKey<T>::with
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:352:9
26: tokio::coop::with_budget
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:99:5
tokio::coop::budget
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:76:5
tokio::runtime::basic_scheduler::Inner<P>::block_on::{{closure}}
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:208:39
27: tokio::runtime::basic_scheduler::enter::{{closure}}
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:299:29
28: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/macros/scoped_tls.rs:61:9
29: tokio::runtime::basic_scheduler::enter
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:299:5
30: tokio::runtime::basic_scheduler::Inner<P>::block_on
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:197:9
31: tokio::runtime::basic_scheduler::InnerGuard<P>::block_on
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:452:9
32: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:157:24
33: tokio::runtime::Runtime::block_on
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/mod.rs:450:46
34: tokio::task::local::LocalSet::block_on
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:477:9
35: actix_rt::runtime::Runtime::block_on
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.1.0/src/runtime.rs:85:9
36: actix_rt::system::SystemRunner::block_on
at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.1.0/src/system.rs:186:9
37: restapi::main
at src/bin/restapi.rs:17:1
38: core::ops::function::FnOnce::call_once
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/ops/function.rs:227:5
39: std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/sys_common/backtrace.rs:125:18
40: std::rt::lang_start::{{closure}}
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:49:18
41: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/ops/function.rs:259:13
std::panicking::try::do_call
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:379:40
std::panicking::try
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:343:19
std::panic::catch_unwind
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panic.rs:431:14
std::rt::lang_start_internal
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:34:21
42: std::rt::lang_start
at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:48:5
43: main
44: __libc_start_main
45: _start
HAOYUatHZ commented
I try again but cannot reproduce.
This time the binaries can start but the data looks weird: kline lose old data & we don't have new blocks