Syncing readonly node often results in status code error
e00dan opened this issue · 3 comments
e00dan commented
I often get status code parsing error
godwoken_1 | [2021-12-18T17:46:33Z INFO gw_block_producer::poller] known last tx hash, skip 4832dfcf5e8abda098d7ff3b29319a397327e032877f7f565cdeae7934abe034
godwoken_1 | [2021-12-18T17:46:35Z INFO gw_chain::chain] sync new block #74362 success
godwoken_1 | [2021-12-18T17:46:35Z INFO gw_block_producer::poller] known last tx hash, skip 823a671f530fdfc3584cd0e7cfa41d53cda95a420277fb28787f6cb9bcc1d62d
godwoken_1 | [2021-12-18T17:46:37Z INFO gw_chain::chain] sync new block #74363 success
godwoken_1 | [2021-12-18T17:46:37Z INFO gw_block_producer::poller] known last tx hash, skip 68359d22410bb4fcd37bad755689769e1ee56b6d8639e358270e061792e2deac
godwoken_1 | thread 'smol-5' panicked at 'Could not convert into a valid `StatusCode`: Invalid status code', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/http-types-2.12.0/src/response.rs:63:14
godwoken_1 | stack backtrace:
godwoken_1 | 0: 0x55c99752c9b0 - std::backtrace_rs::backtrace::libunwind::trace::h34055254b57d8e79
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
godwoken_1 | 1: 0x55c99752c9b0 - std::backtrace_rs::backtrace::trace_unsynchronized::h8f1e3fbd9afff6ec
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
godwoken_1 | 2: 0x55c99752c9b0 - std::sys_common::backtrace::_print_fmt::h3a99a796b770c360
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:67:5
godwoken_1 | 3: 0x55c99752c9b0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h32d1f94a80615d18
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:46:22
godwoken_1 | 4: 0x55c997550d2c - core::fmt::write::h306731c068f7162c
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/fmt/mod.rs:1110:17
godwoken_1 | 5: 0x55c997526185 - std::io::Write::write_fmt::hd2fa90334eee2a21
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/io/mod.rs:1588:15
godwoken_1 | 6: 0x55c99752eefb - std::sys_common::backtrace::_print::h5abaa2601a852287
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:49:5
godwoken_1 | 7: 0x55c99752eefb - std::sys_common::backtrace::print::h8d81445442bb638f
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:36:9
godwoken_1 | 8: 0x55c99752eefb - std::panicking::default_hook::{{closure}}::hcfe804496a9fa747
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:208:50
godwoken_1 | 9: 0x55c99752e9d1 - std::panicking::default_hook::hbea8e3ccf2ba8901
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:225:9
godwoken_1 | 10: 0x55c99752f5c4 - std::panicking::rust_panic_with_hook::h7ee9e1a2d0f8975a
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:622:17
godwoken_1 | 11: 0x55c99752f0a7 - std::panicking::begin_panic_handler::{{closure}}::h8ab3b4491718b2c7
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:519:13
godwoken_1 | 12: 0x55c99752ceac - std::sys_common::backtrace::__rust_end_short_backtrace::hd489062ffa586a9f
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys_common/backtrace.rs:141:18
godwoken_1 | 13: 0x55c99752f009 - rust_begin_unwind
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
godwoken_1 | 14: 0x55c996739ce1 - core::panicking::panic_fmt::hca6330e3e14086b4
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/panicking.rs:92:14
godwoken_1 | 15: 0x55c996739dd3 - core::result::unwrap_failed::h6160d2b4bfbb0e87
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/result.rs:1355:5
godwoken_1 | 16: 0x55c996ee8d36 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hf48c5e0886c50ad7
godwoken_1 | 17: 0x55c9968d62d6 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h6eca5ab2e2529915
godwoken_1 | 18: 0x55c9968d6dd6 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h7695cd6265be8ed5
godwoken_1 | 19: 0x55c9968dab4b - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h9131097b288f5770
godwoken_1 | 20: 0x55c996916b46 - gw_block_producer::poller::ChainUpdater::update_single::{{closure}}::h884924c702b8fb1d
godwoken_1 | 21: 0x55c9969246fd - gw_block_producer::runner::poll_loop::{{closure}}::h9508ee78122bc75d
godwoken_1 | 22: 0x55c9968dc7eb - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h9222195d3f8113d8
godwoken_1 | 23: 0x55c996a4b2ac - async_task::raw::RawTask<F,T,S>::run::hc2dc0e997da06eae
godwoken_1 | 24: 0x55c996fb5600 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h2d5c9ca6a2c8e888
godwoken_1 | 25: 0x55c996fb5731 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h5d8f10918f2408b9
godwoken_1 | 26: 0x55c996fb48d1 - async_io::driver::block_on::hf2300471d14777c4
godwoken_1 | 27: 0x55c996fb4097 - smol::spawn::spawn::GLOBAL::{{closure}}::{{closure}}::hf82b93cf38fdea7c
godwoken_1 | 28: 0x55c996fb3d26 - std::sys_common::backtrace::__rust_begin_short_backtrace::h589886bca5805b01
godwoken_1 | 29: 0x55c996fb335b - core::ops::function::FnOnce::call_once{{vtable.shim}}::h06680d20e2a16ee8
godwoken_1 | 30: 0x55c997533a87 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h7ece6cfefaff1005
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/alloc/src/boxed.rs:1575:9
godwoken_1 | 31: 0x55c997533a87 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hb8b48e55c21f193e
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/alloc/src/boxed.rs:1575:9
godwoken_1 | 32: 0x55c997533a87 - std::sys::unix::thread::Thread::new::thread_start::h8c7c4450dba62914
godwoken_1 | at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/sys/unix/thread.rs:71:17
godwoken_1 | 33: 0x7f1153153450 - start_thread
godwoken_1 | 34: 0x7f11528d0d53 - clone
It seems error is happening in:
godwoken_1 | 20: 0x55c996916b46 - gw_block_producer::poller::ChainUpdater::update_single::{{closure}}::h884924c702b8fb1d
godwoken_1 | 21: 0x55c9969246fd - gw_block_producer::runner::poll_loop::{{closure}}::h9508ee78122bc75d
The worst part of this error is that it doesn't kill the process so it doesn't auto restart. The process is hanging with the critical error. I think critical error should terminate process so eg. restart: always in docker-compose can restart the process. This is my docker compose file (it contains already RUST_LOG: info,http_client=debug):
version: '3.8'
services:
postgres:
image: postgres:alpine
ports:
- 5432:5432
environment:
POSTGRES_USER: user
POSTGRES_DB: lumos
POSTGRES_PASSWORD: password
volumes:
- ../cache/activity/postgres-data:/var/lib/postgresql/data
- ../docker/postgres/web3-indexer-sql/create_tables.sql:/docker-entrypoint-initdb.d/create_tables.sql
redis:
image: redis:bullseye
user: redis:redis
ports:
- 6379:6379
volumes:
- ../cache/activity/redis-data:/data
godwoken:
image: nervos/godwoken-prebuilds:v0.7.2
restart: always
volumes:
- ../testnet-godwoken-readonly-config.toml:/deploy/config.toml
- ../block-producer-private-key.txt:/deploy/pk
- ../cache/activity/store.db:/store.db
environment:
RUST_BACKTRACE: full
RUST_LOG: info,http_client=debug
command: godwoken run -c /deploy/config.toml
ports:
- 8119:8119
depends_on:
- postgres
web3:
image: nervos/godwoken-web3-prebuilds:v0.8.1-rc2
volumes:
- ../web3.env:/godwoken-web3/packages/api-server/.env
working_dir: /godwoken-web3
command: yarn workspace @godwoken-web3/api-server start
ports:
- 8024:8024
depends_on:
- redis
- postgres
- godwoken
This is full repository: https://github.com/Kuzirashi/nervos-godwoken-testnet-node
e00dan commented
I have set more details logs with:
RUST_LOG: info,http_client=debug,gw_block_producer=debug,gw_rpc_client=debug,async_jsonrpc_client=debug
If I hit the error again I'll post more complete logs.
jjyr commented
Can you set RUST_LOG=debug and reproduce this error? The debugging info can help us to locate the wrong RPC call.
e00dan commented
I think it was problem with Layer 1 node. I changed Layer 1 node and managed to sync 200k blocks.