palomachain/paloma

BUG: Validator is showing in active set, but not signing any blocks after upgrade to Paloma v1.13.3

taariq opened this issue · 4 comments

taariq commented

What is happening?

Section description Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.

BUG: Validator is showing in active set, but not signing any blocks after upgrade to Paloma v1.13.3
Paloma Logs:
https://gist.github.com/pcheliniy/5de28dfdeb927073d965456ce474d72a paloma logs

Pigeon Logs:
https://gist.github.com/pcheliniy/2f1d4d0f4dbb8170f69de91ef4beead7 pigeon logs

Paloma and pigeon versions and logs

Section description Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.

Paloma: v1.13.3
commit: 1ba2ba7

Pigeon: 1.11.0

How to reproduce?

Section description Please write detailed steps of what you were doing for this bug to appear.

Not able to reproduce. Validator shows as active: https://paloma.explorers.guru/validator/palomavaloper1p4r04ek0jnngegmw76rpx66xu9ep0kuzvcwxyy

But is not signing.

What is the expected behaviour?

Section description If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.

Active validators must sign.

byte-bandit commented

Looks like they're continuously timing out. It might be related to the hardware used being too weak.

pcheliniy commented

it's dedicated host
memory 128GB
CPU AMD Ryzen 9 5950X 16-Core Processor
2 x 2TB nvme disks (raid0)

systemd-detect-virt
none

cat /proc/cpuinfo | grep 'model name' | head -1
model name      : AMD Ryzen 9 5950X 16-Core Processor

cat /proc/meminfo | head -3
MemTotal:       131821124 kB
MemFree:          848136 kB
MemAvailable:   120575208 kB

Load

mpstat
13:40:53     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
13:40:53     all    2.37    6.05    1.18    0.49    0.00    0.08    0.00    0.00    0.00   89.82

iostat
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1        530.20  10355.55     0.37   0.07    0.17    19.53  101.81  10347.38   130.25  56.13    0.61   101.64    0.42   1281.75     0.00   0.00    0.13  3021.64    0.00    0.00    0.15   7.25
nvme1n1        529.10  10339.99     0.46   0.09    0.17    19.54   78.64   9498.98   129.32  62.18    0.71   120.78    0.43   1299.59     0.00   0.00    0.13  2995.38    0.00    0.00    0.15   7.27
taariq commented

Closing as fixed by @pcheliniy

pcheliniy commented

It was a my mistake and not really a problem, I messed up with node keys, during all this cleanups / inits.