error : thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()`
Closed this issue · 4 comments
hellower commented
First of all, thank you very much for a great open source product
I am currently using pgbouncer, but I am testing various ways to see if it can be replaced with pgcat.
There was no error at all in haproxy+pgbouncer in the previous setting, and after replacing it with pgcat, pgcat continues to occur.
(I keep getting error messages, but pgcat seems to be working fine.)
I found out that the pgcat error below was caused by this part of the haproxy "...tnt-worker1:6432..." above after a long trace.
-
env
os: Ubuntu 20.04.6 LTS
pgcat 1.1.1 (port: 6432)
postgresql 15.4 (port : 9081)
HAProxy version 2.8.1-a90123a 2023/07/03 - https://haproxy.org/ -
haproxy configuration
.......................................
listen cluster-rw-poolport
mode tcp
option tcplog
bind *:5000
timeout client 30m
timeout connect 4s
timeout server 30m
timeout check 5s
option httpchk OPTIONS /read-write # /primary ex) curl -sv http://tnt-worker1:8008/replica | jq .
http-check expect status 200
default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
server tnt-worker1 tnt-worker1:6432 maxconn 100 check port 8008 ############# HERE !!!! #############
server tnt-worker2 tnt-worker2:6432 maxconn 100 check port 8008 # patroni rest-api 8008 ############# HERE !!!! #############
.......................................
- PGCAT error
2023-08-23T23:06:10.443007Z INFO pgcat: Config autoreloader: 86400000 ms
2023-08-23T23:06:10.443044Z INFO pgcat::stats: Events reporter started
2023-08-23T23:06:10.443144Z INFO pgcat: Waiting for clients
2023-08-23T23:06:10.442995Z INFO pgcat::pool: Creating a new server connection Address { id: 9, host: "localhost", port: 9081, shard: 0, database: "postgres", role: Replica, replica_number: 0, address_index: 1, username: "monexp", pool_name: "tx_monexp", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false } }
2023-08-23T23:06:10.443196Z INFO pgcat::pool: Creating a new server connection Address { id: 8, host: "localhost", port: 9081, shard: 0, database: "postgres", role: Primary, replica_number: 0, address_index: 0, username: "monexp", pool_name: "tx_monexp", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false } }
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-08-23T23:06:11.365586Z INFO pgcat::server::cleanup: Server returned with session state altered, discarding state (SET: true, PREPARE: false) for application TNT-CLUSTER#12
2023-08-23T23:06:12.421295Z INFO pgcat::server::cleanup: Server returned with session state altered, discarding state (SET: true, PREPARE: false) for application TARANTULA:0.0.0.0:9002/3.4.71@local:b
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
- PGCAT full error message (export RUST_BACKTRACE=full)
.......
2023-08-23T23:06:38.795991Z INFO pgcat::pool: Creating a new server connection Address { id: 9, host: "localhost", port: 9081, shard: 0, database: "postgres", role: Replica, replica_number: 0, address_index: 1, username: "monexp", pool_name: "tx_monexp", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false } }
2023-08-23T23:06:38.795465Z INFO pgcat::stats: Events reporter started
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
stack backtrace:
0: 0x560485a9d861 - std::backtrace_rs::backtrace::libunwind::trace::h782cc21a5acaf6cb
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x560485a9d861 - std::backtrace_rs::backtrace::trace_unsynchronized::hc579eb24ab204515
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x560485a9d861 - std::sys_common::backtrace::_print_fmt::h7223525cfdbacda2
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:65:5
3: 0x560485a9d861 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbd7d55b7108d2ab8
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:44:22
4: 0x560485ac506f - core::fmt::rt::Argument::fmt::hb4f4a02b9bd9dd49
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/fmt/rt.rs:138:9
5: 0x560485ac506f - core::fmt::write::h6d54cd7c9e155ec5
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/fmt/mod.rs:1094:21
6: 0x560485a9aa41 - std::io::Write::write_fmt::h6a453a71c692f63b
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/io/mod.rs:1713:15
7: 0x560485a9d675 - std::sys_common::backtrace::_print::h1cbaa8b42678f928
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:47:5
8: 0x560485a9d675 - std::sys_common::backtrace::print::h4ddf81241a51b337
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:34:9
9: 0x560485a9e997 - std::panicking::default_hook::{{closure}}::hff91f1f484ade5cd
10: 0x560485a9e784 - std::panicking::default_hook::h21f14afd59f7aef9
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:288:9
11: 0x560485a9ee4c - std::panicking::rust_panic_with_hook::h45f66047b14c555c
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:705:13
12: 0x560485a9ed47 - std::panicking::begin_panic_handler::{{closure}}::h49d1a88ef0908eb4
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:597:13
13: 0x560485a9dc96 - std::sys_common::backtrace::__rust_end_short_backtrace::hccebf9e57f8cc425
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:151:18
14: 0x560485a9ea92 - rust_begin_unwind
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:593:5
15: 0x560485212d63 - core::panicking::panic_fmt::h54ec9d0e3180a83d
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:67:14
16: 0x560485213203 - core::result::unwrap_failed::h1cd730365d65235f
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/result.rs:1651:5
17: 0x560485439e9d - pgcat::client::client_entrypoint::{{closure}}::h152ca8414b8b9198
18: 0x56048547dac9 - pgcat::main::{{closure}}::{{closure}}::h5e6389a6bf4b6203
19: 0x560485472fd1 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h6a20aed4daa98b35
20: 0x5604853541f6 - tokio::runtime::task::core::Core<T,S>::poll::h57ff81af1c9d7d81
21: 0x56048522acae - tokio::runtime::task::harness::Harness<T,S>::poll::h744e0029abcf98ae
22: 0x560485a4b47a - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h9b544f66c4ef6a0e
23: 0x560485a4aa5e - tokio::runtime::scheduler::multi_thread::worker::Context::run::h19de324b23b290bd
24: 0x560485a5bd27 - tokio::runtime::context::scoped::Scoped<T>::set::he66079070aa2bfdd
25: 0x560485a4dd15 - tokio::runtime::context::runtime::enter_runtime::hbc778c7fd1b00890
26: 0x560485a4a15c - tokio::runtime::scheduler::multi_thread::worker::run::h6f9959fd400d1aac
27: 0x560485a58f9b - tokio::runtime::task::core::Core<T,S>::poll::he638b91f0b1532e1
28: 0x560485a44f5f - tokio::runtime::task::harness::Harness<T,S>::poll::h70ce99c790196f83
29: 0x560485a632f6 - tokio::runtime::blocking::pool::Inner::run::hbb956397631ab4b3
30: 0x560485a46875 - std::sys_common::backtrace::__rust_begin_short_backtrace::h192e29676f1f0e03
31: 0x560485a6fbb9 - core::ops::function::FnOnce::call_once{{vtable.shim}}::he30264600efd9a83
32: 0x560485aa1875 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::he3e5dbdfabe0b668
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/alloc/src/boxed.rs:1985:9
33: 0x560485aa1875 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h246f7c7964633611
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/alloc/src/boxed.rs:1985:9
34: 0x560485aa1875 - std::sys::unix::thread::Thread::new::thread_start::hadf9e3501ff0df23
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys/unix/thread.rs:108:17
35: 0x7fabc7566609 - start_thread
36: 0x7fabc7336133 - clone
37: 0x0 - <unknown>
2023-08-23T23:06:39.197253Z INFO pgcat::server::cleanup: Server returned with session state altered, discarding state (SET: true, PREPARE: false) for application TNT-CLUSTER#13
2023-08-23T23:06:40.239704Z INFO pgcat::server::cleanup: Server returned with session state altered, discarding state (SET: true, PREPARE: false) for application TARANTULA:0.0.0.0:9001/3.4.71@local:b
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
stack backtrace:
0: 0x560485a9d861 - std::backtrace_rs::backtrace::libunwind::trace::h782cc21a5acaf6cb
at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/
hellower commented
Can I change the 120th line in the same way? (1.1.2-dev src/client.rs source . .line 120)
109 /// Client entrypoint.
110 pub async fn client_entrypoint(
111 mut stream: TcpStream,
112 client_server_map: ClientServerMap,
113 shutdown: Receiver<()>,
114 drain: Sender,
115 admin_only: bool,
116 tls_certificate: Option,
117 log_client_connections: bool,
118 ) -> Result<(), Error> {
119 // Figure out if the client wants TLS or not.
120 let addr = stream.peer_addr().unwrap(); <======== here!
...
let addr = match stream.peer_addr() {
Ok(addr) => addr,
Err(err) => {
return Err(Error::SocketError(format!(
"Failed to get peer address: {:?}",
err
)));
}
};
hellower commented
I've confirmed that it's a haporoxy problem and not a pgcat problem.
levkk commented
A PR to replace the unwrap with an error would still be welcome. An error is better than a panic.
hellower commented
Thanks to Zain Kabani!