Blockchain node stop syncing. receiving a lot of errors about invalid transactions
Closed this issue · 10 comments
APshenkin commented
We updated blockchain-node to 1.1.66 two days ago before todays hard fork
But starting 18:00 UTC time syncronization stops. Get a lot of errors in logs for all nodes
2022-06-30 19:25:43.331 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="1DSpewzLZ6PQGXWUP8zvAvMnZXsopiB4nSyQic8NopdwmuYie5" challenger="blurry-fern-dog" onion="12hA6TED9axC8R7ddbbCLdz9KEnNkubk66Q1qFW48vXrP2drMQJ" path:
type=element challengee: upbeat-chiffon-buffalo, receipt: type=receipt gateway: upbeat-chiffon-buffalo timestamp: 1656611776163321440 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 19
witnesses: type=witness gateway: "skinny-porcelain-tarantula" timestamp: 1656611776733936813 signal: -101 snr: -5.0 freq 487.5
type=witness gateway: "breezy-walnut-barracuda" timestamp: 1656611776716537276 signal: -101 snr: -5.400000095367432 freq 487.5
type=witness gateway: "bubbly-dijon-tarantula" timestamp: 1656611776705227730 signal: -107 snr: -6.699999809265137 freq 487.5
type=witness gateway: "oblong-strawberry-bear" timestamp: 1656611776711994130 signal: -100 snr: -6.199999809265137 freq 487.5
type=witness gateway: "bright-pearl-corgi" timestamp: 1656611776699924883 signal: -101 snr: -5.099999904632568 freq 487.5
type=witness gateway: "short-midnight-swift" timestamp: 1656611776715537536 signal: -102 snr: -5.599999904632568 freq 487.5
2022-06-30 19:25:43.333 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="1SCb5x2Sd972h1PNKNCBTeTKyUrDeuBYjqzBhKmWgrg3ujZM7N" challenger="big-glass-gerbil" onion="1gpTtkCEUD8TLTfzRTZsDmeFgAPwzTgUq8hgoWE1NUt7HyJ7aX" path:
type=element challengee: deep-hazel-python, receipt: type=receipt gateway: deep-hazel-python timestamp: 1656611609073139566 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 16
witnesses: type=witness gateway: "decent-midnight-oyster" timestamp: 1656611611797776427 signal: -125 snr: -9.5 freq 868.2999877929688
type=witness gateway: "narrow-stone-rattlesnake" timestamp: 1656611611728820043 signal: -135 snr: -17.0 freq 868.2999877929688
type=witness gateway: "fresh-canvas-gecko" timestamp: 1656611611859472733 signal: -119 snr: -17.799999237060547 freq 868.2999877929688
type=witness gateway: "bent-slate-bison" timestamp: 1656611611703473731 signal: -121 snr: -8.800000190734863 freq 868.2999877929688
type=witness gateway: "tart-pine-chimpanzee" timestamp: 1656611611704493909 signal: -120 snr: -13.800000190734863 freq 868.2999877929688
type=witness gateway: "jolly-fuchsia-moose" timestamp: 1656611611712605614 signal: -121 snr: -16.5 freq 868.2999877929688
type=witness gateway: "zealous-caramel-piranha" timestamp: 1656611611704078295 signal: -123 snr: -9.5 freq 868.2999877929688
type=witness gateway: "wild-tangelo-sidewinder" timestamp: 1656611611732078980 signal: -124 snr: -8.800000190734863 freq 868.2999877929688
type=witness gateway: "witty-coal-quail" timestamp: 1656611611715400469 signal: -21 snr: 7.5 freq 868.2999877929688
type=witness gateway: "dancing-dijon-wallaby" timestamp: 1656611611813271357 signal: -120 snr: -8.199999809265137 freq 868.2999877929688
type=witness gateway: "tame-eggshell-corgi" timestamp: 1656611610077991139 signal: -50 snr: 7.5 freq 868.2999877929688
type=witness gateway: "proud-laurel-carp" timestamp: 1656611611720149587 signal: -122 snr: -11.0 freq 868.2999877929688
type=witness gateway: "small-garnet-python" timestamp: 1656611611686151389 signal: -116 snr: -3.200000047683716 freq 868.2999877929688
type=witness gateway: "original-orchid-mammoth" timestamp: 1656611611681383122 signal: -120 snr: -13.800000190734863 freq 868.2999877929688
2022-06-30 19:25:43.350 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="123Wi3LTSLoQDevviZhX5BpedKCwyqg8uEBx2mCxbY1ykJ1V8rn" challenger="radiant-infrared-frog" onion="1QP3M342c5xviyQ9LctNcnLQEMxMwi8soCr8ZgwyRt4TSkhKp4" path:
type=element challengee: shallow-merlot-seahorse, receipt: type=receipt gateway: shallow-merlot-seahorse timestamp: 1656611825268313732 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 28
witnesses: type=witness gateway: "tame-fossilized-tarantula" timestamp: 1656611826700517015 signal: -126 snr: -10.5 freq 904.0999755859375
type=witness gateway: "helpful-mint-wolverine" timestamp: 1656611826698507512 signal: -113 snr: 0.0 freq 904.0999755859375
type=witness gateway: "blunt-flaxen-hippo" timestamp: 1656611826710092207 signal: -122 snr: -7.199999809265137 freq 904.0999755859375
type=witness gateway: "salty-pear-blackbird" timestamp: 1656611826697023499 signal: -119 snr: -2.200000047683716 freq 904.0999755859375
type=witness gateway: "modern-black-caribou" timestamp: 1656611826693168282 signal: -117 snr: -11.5 freq 904.0999755859375
type=witness gateway: "wide-cinnamon-newt" timestamp: 1656611826678937001 signal: -106 snr: 0.0 freq 904.0999755859375
type=witness gateway: "jovial-concrete-nightingale" timestamp: 1656611826695618551 signal: -94 snr: -0.5 freq 904.0999755859375
type=witness gateway: "fantastic-rosewood-stallion" timestamp: 1656611826689361031 signal: -100 snr: 4.800000190734863 freq 904.0999755859375
type=witness gateway: "dazzling-clear-huskie" timestamp: 1656611826694993378 signal: -125 snr: -12.0 freq 904.0999755859375
type=witness gateway: "damaged-purple-rooster" timestamp: 1656611826686999444 signal: -126 snr: -11.800000190734863 freq 904.0999755859375
type=witness gateway: "jumpy-magenta-rattlesnake" timestamp: 1656611826800220409 signal: -128 snr: -12.199999809265137 freq 904.0999755859375
type=witness gateway: "shallow-paisley-walrus" timestamp: 1656611824385160818 signal: -120 snr: -7.0 freq 904.0999755859375
type=witness gateway: "quiet-currant-haddock" timestamp: 1656611826688421713 signal: -123 snr: -13.0 freq 904.0999755859375
type=witness gateway: "scrawny-sapphire-wolf" timestamp: 1656611826715789299 signal: -126 snr: -13.0 freq 904.0999755859375
2022-06-30 19:25:43.352 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="1LL6GuWhG2gr7spYaP3tYbt8cbDekUtwMWYxcnoehvMyRFiMJX" challenger="plain-seafoam-gecko" onion="12vgjStdoyFcDCVV9MMw7meajvakFVJk568DLuEduSEhkSuLCXH" path:
type=element challengee: joyous-lavender-moose, receipt: type=receipt gateway: joyous-lavender-moose timestamp: 1656611760359299761 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 19
witnesses: type=witness gateway: "wide-olive-dove" timestamp: 1656611760955250679 signal: -98 snr: 9.0 freq 486.5
type=witness gateway: "unique-wool-vulture" timestamp: 1656611760937445042 signal: -100 snr: 6.5 freq 486.5
type=witness gateway: "curly-daisy-cow" timestamp: 1656611760947828336 signal: -98 snr: 9.0 freq 486.5
type=witness gateway: "eager-sable-seagull" timestamp: 1656611760949852426 signal: -95 snr: 10.0 freq 486.5
type=witness gateway: "furry-daisy-fox" timestamp: 1656611760935356401 signal: -96 snr: 9.5 freq 486.5
type=witness gateway: "wild-candy-tortoise" timestamp: 1656611761011933173 signal: -99 snr: 8.199999809265137 freq 486.5
type=witness gateway: "prehistoric-brunette-mustang" timestamp: 1656611760949550748 signal: -112 snr: -0.5 freq 486.5
type=witness gateway: "dandy-burlap-otter" timestamp: 1656611760941818313 signal: -94 snr: 9.800000190734863 freq 486.5
type=witness gateway: "great-lead-shark" timestamp: 1656611760937499012 signal: -102 snr: 7.5 freq 486.5
type=witness gateway: "hollow-myrtle-jellyfish" timestamp: 1656611760953907568 signal: -87 snr: 11.0 freq 486.5
type=witness gateway: "colossal-citron-baboon" timestamp: 1656611760951816145 signal: -90 snr: 10.5 freq 486.5
type=witness gateway: "brisk-strawberry-snail" timestamp: 1656611760940155869 signal: -96 snr: 9.0 freq 486.5
type=witness gateway: "ancient-steel-rook" timestamp: 1656611760962844576 signal: -106 snr: 4.0 freq 486.5
type=witness gateway: "atomic-vinyl-ferret" timestamp: 1656611760946258613 signal: -84 snr: 11.0 freq 486.5
2022-06-30 19:25:43.354 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="1TmV62a4k9gdUAAd7ZMy9hR3UNwP7WZ5W9x2Ex7m9M1ATiUFhh" challenger="fluffy-lead-jellyfish" onion="1gpe9rfmbk4Xt8Mrs25cSrH7Z7E2CPBF6fNmmG2MXNATrJxp3p" path:
type=element challengee: strong-hickory-bear, receipt: type=receipt undefined
witnesses: type=witness gateway: "square-topaz-pangolin" timestamp: 1656611768957356245 signal: -107 snr: 1.5 freq 868.0999755859375
type=witness gateway: "eager-khaki-antelope" timestamp: 1656611774406760463 signal: -139 snr: -14.0 freq 868.0999755859375
type=witness gateway: "interesting-metal-mink" timestamp: 1656611763702500516 signal: -115 snr: -7.800000190734863 freq 868.0999755859375
type=witness gateway: "damp-brown-ostrich" timestamp: 1656611781082579381 signal: -114 snr: -4.5 freq 868.0999755859375
type=witness gateway: "colossal-bamboo-robin" timestamp: 1656611774551995124 signal: -123 snr: -5.199999809265137 freq 868.0999755859375
type=witness gateway: "interesting-umber-crocodile" timestamp: 1656611761392375293 signal: -113 snr: -8.0 freq 868.0999755859375
type=witness gateway: "sharp-crimson-fish" timestamp: 1656611766853918516 signal: -132 snr: -8.800000190734863 freq 868.0999755859375
type=witness gateway: "sunny-sandstone-spider" timestamp: 1656611774719612345 signal: -119 snr: -6.199999809265137 freq 868.0999755859375
type=witness gateway: "acidic-sandstone-tadpole" timestamp: 1656611781291166806 signal: -99 snr: 14.0 freq 868.0999755859375
type=witness gateway: "fresh-yellow-cougar" timestamp: 1656611782903844518 signal: -111 snr: 4.800000190734863 freq 868.0999755859375
type=witness gateway: "careful-cornflower-scorpion" timestamp: 1656611763797598029 signal: -118 snr: -6.0 freq 868.0999755859375
type=witness gateway: "gentle-nylon-toad" timestamp: 1656611795309289980 signal: -126 snr: -8.800000190734863 freq 868.0999755859375
type=witness gateway: "faint-marmalade-swan" timestamp: 1656611787783613770 signal: -123 snr: 3.200000047683716 freq 868.0999755859375
type=witness gateway: "soft-wool-halibut" timestamp: 1656611777258227734 signal: -106 snr: 7.0 freq 868.0999755859375
2022-06-30 19:25:43.356 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="12HhgdDmFinjtMbtLP5rbkwJCE2AupajRvPNchYgXxa6QHivkh3" challenger="fluffy-lead-jellyfish" onion="1Eua1WWDgGavv5EYDURa61gjNYHzeKbpnp1PNjhjQdMrFi8jic" path:
type=element challengee: clean-coffee-skunk, receipt: type=receipt gateway: clean-coffee-skunk timestamp: 1656611760736038445 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 32
witnesses: type=witness gateway: "zesty-flaxen-wren" timestamp: 1656611761236677223 signal: -118 snr: -5.0 freq 904.0999755859375
type=witness gateway: "zesty-emerald-baboon" timestamp: 1656611761221871626 signal: -103 snr: 7.199999809265137 freq 904.0999755859375
type=witness gateway: "swift-misty-hyena" timestamp: 1656611761216076382 signal: -114 snr: -5.5 freq 904.0999755859375
type=witness gateway: "fierce-pineapple-aardvark" timestamp: 1656611761252700785 signal: -113 snr: -5.5 freq 904.0999755859375
type=witness gateway: "noisy-candy-camel" timestamp: 1656611761220512722 signal: -107 snr: 1.5 freq 904.0999755859375
type=witness gateway: "puny-red-platypus" timestamp: 1656611761291001074 signal: -122 snr: -14.0 freq 904.0999755859375
type=witness gateway: "rich-ultraviolet-unicorn" timestamp: 1656611761223901712 signal: -118 snr: -3.799999952316284 freq 904.0999755859375
type=witness gateway: "modern-rainbow-pike" timestamp: 1656611761217254711 signal: -120 snr: -11.5 freq 904.0999755859375
type=witness gateway: "quaint-lemonade-deer" timestamp: 1656611761215674646 signal: -112 snr: -5.199999809265137 freq 904.0999755859375
type=witness gateway: "bubbly-orchid-raccoon" timestamp: 1656611759699105547 signal: -116 snr: -3.0 freq 904.0999755859375
2022-06-30 19:25:43.357 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="1E9zpLDMeTABMLYoqnUApDYhHGUTFTpnn9Z5MaDVquAhovwKAG" challenger="elegant-pebble-crow" onion="1oPYrDtvdmibgSDgrjcXfW23dF4FDiacwGB9hRFhpikWLxDgtG" path:
type=element challengee: cuddly-banana-orangutan, receipt: type=receipt gateway: cuddly-banana-orangutan timestamp: 1656611759802755026 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 16
witnesses: type=witness gateway: "ripe-blonde-frog" timestamp: 1656611761420364884 signal: -114 snr: -6.0 freq 868.2999877929688
type=witness gateway: "sharp-menthol-sloth" timestamp: 1656611763447684309 signal: -124 snr: -17.5 freq 868.2999877929688
type=witness gateway: "daring-cherry-mantaray" timestamp: 1656611761950389108 signal: -85 snr: 6.800000190734863 freq 868.2999877929688
type=witness gateway: "pet-garnet-stork" timestamp: 1656611763376991686 signal: -130 snr: -20.200000762939453 freq 868.2999877929688
type=witness gateway: "glamorous-citron-crab" timestamp: 1654284122237808855 signal: -111 snr: -12.5 freq 868.2999877929688
type=witness gateway: "urban-tan-okapi" timestamp: 1656611763363435853 signal: -124 snr: -21.0 freq 868.2999877929688
type=witness gateway: "dizzy-grey-leopard" timestamp: 1656611763485973039 signal: -128 snr: -18.0 freq 868.2999877929688
type=witness gateway: "mysterious-flaxen-cougar" timestamp: 1656611763368754087 signal: -128 snr: -17.0 freq 868.2999877929688
type=witness gateway: "brisk-peanut-tuna" timestamp: 1656611763404004821 signal: -103 snr: -18.799999237060547 freq 868.2999877929688
type=witness gateway: "glorious-smoke-gazelle" timestamp: 1656611763377853869 signal: -129 snr: -22.5 freq 868.2999877929688
type=witness gateway: "howling-green-dragon" timestamp: 1656611763361324734 signal: -106 snr: 2.0 freq 868.2999877929688
type=witness gateway: "generous-wooden-peacock" timestamp: 1656611763390641219 signal: -101 snr: 4.800000190734863 freq 868.2999877929688
type=witness gateway: "energetic-cotton-nightingale" timestamp: 1656611763488954145 signal: -120 snr: -13.0 freq 868.2999877929688
type=witness gateway: "perfect-ebony-jellyfish" timestamp: 1656611762796623552 signal: -128 snr: -18.799999237060547 freq 868.2999877929688
2022-06-30 19:25:43.377 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="1CRLeBstKwzdzqhLR9rPUqNfMF7kV3F8yGvAQd5AmdAciEoSHa" challenger="unique-hazelnut-zebra" onion="1wn8XwXdtWVPKvgF2p5sXYULhdDpfqHNBHUbuhGgmQ1uLodW8d" path:
type=element challengee: nice-scarlet-fly, receipt: type=receipt gateway: nice-scarlet-fly timestamp: 1656611822741589534 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 34
witnesses: type=witness gateway: "wobbly-brown-ferret" timestamp: 1656611824679261053 signal: -121 snr: -7.0 freq 904.7000122070313
type=witness gateway: "striped-caramel-crane" timestamp: 1656611824686879707 signal: -113 snr: -6.800000190734863 freq 904.7000122070313
type=witness gateway: "recumbent-chrome-cougar" timestamp: 1656611824693531287 signal: -104 snr: 7.800000190734863 freq 904.7000122070313
type=witness gateway: "rapid-pewter-dolphin" timestamp: 1656611824688259940 signal: -111 snr: 1.5 freq 904.7000122070313
type=witness gateway: "pet-wooden-deer" timestamp: 1656611824777102322 signal: -125 snr: -10.800000190734863 freq 904.7000122070313
type=witness gateway: "tall-charcoal-fish" timestamp: 1656611824677659736 signal: -117 snr: -11.199999809265137 freq 904.7000122070313
type=witness gateway: "curly-cream-osprey" timestamp: 1656611824688406668 signal: -113 snr: -6.0 freq 904.7000122070313
type=witness gateway: "thankful-concrete-tarantula" timestamp: 1656611824694625340 signal: -128 snr: -10.199999809265137 freq 904.7000122070313
type=witness gateway: "scrawny-scarlet-okapi" timestamp: 1656611823103324505 signal: -111 snr: -4.0 freq 904.7000122070313
type=witness gateway: "damp-coal-swan" timestamp: 1656611824693932159 signal: -120 snr: -9.0 freq 904.7000122070313
type=witness gateway: "damaged-umber-beetle" timestamp: 1656611824691228341 signal: -113 snr: -5.0 freq 904.7000122070313
type=witness gateway: "soft-bone-halibut" timestamp: 1656611824680595415 signal: -80 snr: 13.0 freq 904.7000122070313
type=witness gateway: "spare-brown-starfish" timestamp: 1656611824681718505 signal: -122 snr: -13.800000190734863 freq 904.7000122070313
type=witness gateway: "tangy-steel-griffin" timestamp: 1656611824724092333 signal: -76 snr: 13.0 freq 904.7000122070313
2022-06-30 19:25:43.377 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="12UC5raDfjU7D1TfuPfcARDzJKQWQQF2TgnoBdDMoTjH17rrQ6t" challenger="unique-hazelnut-zebra" onion="1rpNA6DMPATKrX2jyuMBKTar21ZnDET7eYJac8JykMPmDkEsJ8" path:
type=element challengee: flaky-porcelain-pangolin, receipt: type=receipt gateway: flaky-porcelain-pangolin timestamp: 1656611797573126175 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 16
witnesses: type=witness gateway: "festive-sand-coyote" timestamp: 1656611801522983094 signal: -123 snr: -9.0 freq 868.5
2022-06-30 19:25:43.379 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="12nFfGWAdJqJGAZ4EAs6tZQLgB5TmidQHZC7bR94UqDyG6jABcr" challenger="unique-hazelnut-zebra" onion="1CqDHPQG2ssgoiDzjMo3js8JDNYsMsLfZ7NsJpHgKdjyDXqnVq" path:
type=element challengee: shaggy-walnut-crocodile, receipt: type=receipt gateway: shaggy-walnut-crocodile timestamp: 1656611844031648342 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 30
witnesses: type=witness gateway: "wobbly-vanilla-jay" timestamp: 1656611845409626666 signal: -109 snr: 3.0 freq 903.9000244140625
type=witness gateway: "harsh-eggplant-nightingale" timestamp: 1656611843412108788 signal: -90 snr: 8.800000190734863 freq 903.9000244140625
type=witness gateway: "soaring-seaweed-swift" timestamp: 1656611845405029834 signal: -116 snr: -10.0 freq 903.9000244140625
type=witness gateway: "obedient-mustard-lion" timestamp: 1656611845414544417 signal: -119 snr: -5.199999809265137 freq 903.9000244140625
type=witness gateway: "attractive-goldenrod-seagull" timestamp: 1656611845418579135 signal: -103 snr: 8.5 freq 903.9000244140625
type=witness gateway: "striped-coffee-osprey" timestamp: 1656611845414398928 signal: -90 snr: 7.800000190734863 freq 903.9000244140625
type=witness gateway: "kind-sage-wolverine" timestamp: 1656611845404225691 signal: -97 snr: 7.199999809265137 freq 903.9000244140625
type=witness gateway: "sweet-peach-pelican" timestamp: 1656611845422313751 signal: -95 snr: 7.199999809265137 freq 903.9000244140625
type=witness gateway: "oblong-hazel-carp" timestamp: 1656611843783645358 signal: -104 snr: 7.0 freq 903.9000244140625
type=witness gateway: "virtual-ebony-fly" timestamp: 1656611845411776897 signal: -96 snr: 7.800000190734863 freq 903.9000244140625
type=witness gateway: "ancient-crepe-beetle" timestamp: 1656611845415875824 signal: -129 snr: -14.199999809265137 freq 903.9000244140625
type=witness gateway: "cold-honey-scorpion" timestamp: 1656611848454865994 signal: -119 snr: -11.199999809265137 freq 903.9000244140625
type=witness gateway: "orbiting-felt-turkey" timestamp: 1656611845382680080 signal: -129 snr: -12.800000190734863 freq 903.9000244140625
type=witness gateway: "basic-ultraviolet-chimpanzee" timestamp: 1656611845412420299 signal: -84 snr: 11.0 freq 903.9000244140625
2022-06-30 19:25:43.380 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="12hWvDKi7GRa2zodQYcK9VzrfY8dFBc7rbf77RVk7pawRmqqC25" challenger="jolly-champagne-dachshund" onion="1294eTNA5bzQJbqLbv5yBBpmw8sgxXxMaKAGEtGifwCAA7aWQsf" path:
type=element challengee: obedient-coal-cougar, receipt: type=receipt gateway: obedient-coal-cougar timestamp: 1656611769961187382 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 29
witnesses: type=witness gateway: "spicy-fuzzy-armadillo" timestamp: 1656611771370136656 signal: -114 snr: 5.199999809265137 freq 918.2000122070313
type=witness gateway: "generous-turquoise-starfish" timestamp: 1656611771303914171 signal: -133 snr: -12.0 freq 918.2000122070313
type=witness gateway: "rare-crepe-turtle" timestamp: 1656611771361154980 signal: -130 snr: -9.5 freq 918.2000122070313
type=witness gateway: "teeny-crimson-monkey" timestamp: 1656611771463136902 signal: -132 snr: -12.0 freq 918.2000122070313
type=witness gateway: "damaged-golden-nightingale" timestamp: 1656611771321322950 signal: -129 snr: -10.0 freq 918.2000122070313
type=witness gateway: "expert-mango-hippo" timestamp: 1656611771338443230 signal: -126 snr: -6.5 freq 918.2000122070313
type=witness gateway: "proper-dijon-blackbird" timestamp: 1656611771341817510 signal: -131 snr: -10.0 freq 918.2000122070313
type=witness gateway: "atomic-hemp-raccoon" timestamp: 1656611771327318809 signal: -124 snr: -5.199999809265137 freq 918.2000122070313
type=witness gateway: "docile-pistachio-bird" timestamp: 1656611771332139186 signal: -120 snr: -1.0 freq 918.2000122070313
type=witness gateway: "long-rose-boa" timestamp: 1656611771338774972 signal: -120 snr: -1.2000000476837158 freq 918.2000122070313
type=witness gateway: "magic-fern-huskie" timestamp: 1656611771291781032 signal: -117 snr: 0.800000011920929 freq 918.2000122070313
type=witness gateway: "handsome-ivory-vulture" timestamp: 1656611771347958362 signal: -115 snr: 4.5 freq 918.2000122070313
type=witness gateway: "gigantic-sangria-crane" timestamp: 1656611771366711512 signal: -128 snr: -7.5 freq 918.2000122070313
type=witness gateway: "brisk-syrup-koala" timestamp: 1656611771362844890 signal: -130 snr: -10.5 freq 918.2000122070313
2022-06-30 19:25:43.392 [warning] <0.3235.0>@blockchain_txn:separate_res:384 invalid txn blockchain_txn_poc_receipts_v2 : {error,receipt_not_in_order} / type=poc_receipts_v2 hash="11BdwAEJPUkkTEoW125zp1nGDgL2AVtHDH3hTjevkgYUvpkqoh" challenger="curved-dijon-porcupine" onion="1Rh2cewNWJQArWnkYyxGLRgWoHYvrqiyYX5cz8gXsxjyhQtaCD" path:
type=element challengee: square-turquoise-crocodile, receipt: type=receipt gateway: square-turquoise-crocodile timestamp: 1656611759294028469 signal: 0 snr: 0.0 freq: 0.0 origin: p2p, tx_power: 16
witnesses: type=witness gateway: "wild-malachite-salmon" timestamp: 1656611761175393854 signal: -127 snr: -17.200000762939453 freq 868.2999877929688
type=witness gateway: "cheesy-marmalade-lemur" timestamp: 1656611761363512011 signal: -134 snr: -18.0 freq 868.2999877929688
type=witness gateway: "interesting-rainbow-bear" timestamp: 1656611762847869693 signal: -124 snr: -12.800000190734863 freq 868.2999877929688
type=witness gateway: "large-navy-chimpanzee" timestamp: 1656611762861248302 signal: -91 snr: 3.0 freq 868.2999877929688
type=witness gateway: "curved-umber-pike" timestamp: 1656611762854551298 signal: -107 snr: -4.800000190734863 freq 868.2999877929688
type=witness gateway: "smooth-paisley-bison" timestamp: 1656611762857385496 signal: -113 snr: -10.5 freq 868.2999877929688
type=witness gateway: "shallow-raspberry-sloth" timestamp: 1656611762849725294 signal: -133 snr: -18.200000762939453 freq 868.2999877929688
type=witness gateway: "broad-candy-flamingo" timestamp: 1656611761296013542 signal: -117 snr: -19.200000762939453 freq 868.2999877929688
type=witness gateway: "quick-blush-cougar" timestamp: 1656611762847407296 signal: -123 snr: -7.5 freq 868.2999877929688
type=witness gateway: "fresh-golden-mantaray" timestamp: 1656611762885744498 signal: -131 snr: -16.200000762939453 freq 868.2999877929688
type=witness gateway: "cheesy-zinc-baboon" timestamp: 1656611762882015968 signal: -126 snr: -9.5 freq 868.2999877929688
type=witness gateway: "soft-midnight-puma" timestamp: 1656611761817394380 signal: -127 snr: -12.199999809265137 freq 868.2999877929688
type=witness gateway: "basic-tweed-unicorn" timestamp: 1656611762855882462 signal: -132 snr: -16.200000762939453 freq 868.2999877929688
type=witness gateway: "eager-powder-mule" timestamp: 1656611762928385701 signal: -139 snr: -21.0 freq 868.2999877929688
APshenkin commented
tried to recover from latest snapshot, but issue is still exist
madninja commented
tried to recover from latest snapshot, but issue is still exist
you sure you tried? I'm pretty sure you can't load a snapshot on the cli in blockchain-node so you did something else?
APshenkin commented
I did cleanup the state of node to get fresh snapshot
madninja commented
I did cleanup the state of node to get fresh snapshot
how? and what snapshot did it install?
APshenkin commented
how? and what snapshot did it install?
Just moved state to other folder.
Then after docker container reboot
2022-06-30 19:50:23.718 [info] <0.1392.0>@blockchain_worker:do_snap_source_download:1295 Attempting snapshot download from "https://snapshots.helium.wtf/mainnet/snap-1420561.gz", writing to scratch file "/var/data/snap/snap-1420561.gz.scratch"
2022-06-30 19:51:09.053 [info] <0.1329.0>@blockchain:init_blessed_snapshot:2661 Got snapshot for height 1420561 - attempting install
2022-06-30 19:51:09.404 [info] <0.1329.0>@blockchain:init_blessed_snapshot:2669 Snapshot height 1420510, lagging ledger height 1, leading ledger height 1
2022-06-30 19:51:09.406 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 317768
2022-06-30 19:51:09.744 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger:567 loading snapshot into delayed ledger
2022-06-30 19:51:09.745 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 321904
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded consensus_members from snapshot in 310 ms
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 322576
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded election_height from snapshot in 0 ms
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 351088
2022-06-30 19:51:10.377 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded election_epoch from snapshot in 322 ms
2022-06-30 19:51:10.377 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 329688
2022-06-30 19:51:10.698 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:634 loaded delayed_vars from snapshot in 321 ms
2022-06-30 19:51:10.703 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 345096
2022-06-30 19:51:12.085 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:634 loaded threshold_txns from snapshot in 1382 ms
2022-06-30 19:51:12.085 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 369240
2022-06-30 19:51:13.707 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded master_key from snapshot in 1622 ms
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 312488
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded multi_keys from snapshot in 0 ms
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 315336
APshenkin commented
ready to provide full log if it will help
scratch_1624.txt
APshenkin commented
Also if complete the full fresh launch blockchain-node starts infinitely produce logs like this
2022-06-30 21:19:29.274 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7279.0>, Ref: #Ref<0.292262878.1006895107.202831>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.287 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7280.0>, Ref: #Ref<0.292262878.1006895107.202849>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.302 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7281.0>, Ref: #Ref<0.292262878.1007157249.120946>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.316 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7282.0>, Ref: #Ref<0.292262878.1007157252.43104>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.333 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7283.0>, Ref: #Ref<0.292262878.1006895107.202857>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.342 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7284.0>, Ref: #Ref<0.292262878.1006895107.202863>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.351 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7285.0>, Ref: #Ref<0.292262878.1006895107.202875>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.361 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7286.0>, Ref: #Ref<0.292262878.1006895107.202894>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.371 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7287.0>, Ref: #Ref<0.292262878.1006895107.202896>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.379 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7288.0>, Ref: #Ref<0.292262878.1006895107.202903>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.389 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7289.0>, Ref: #Ref<0.292262878.1006895107.202915>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.405 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7290.0>, Ref: #Ref<0.292262878.1006895106.127977>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.418 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7291.0>, Ref: #Ref<0.292262878.1007157249.120984>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.431 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7292.0>, Ref: #Ref<0.292262878.1007157249.121000>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.443 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7293.0>, Ref: #Ref<0.292262878.1006895106.127999>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.461 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7294.0>, Ref: #Ref<0.292262878.1006895106.128017>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.479 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7295.0>, Ref: #Ref<0.292262878.1007157252.43137>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.488 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7296.0>, Ref: #Ref<0.292262878.1007157252.43148>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.504 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7297.0>, Ref: #Ref<0.292262878.1006895107.202935>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.519 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7298.0>, Ref: #Ref<0.292262878.1006895107.202944>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.535 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7299.0>, Ref: #Ref<0.292262878.1006895107.202963>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.550 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7300.0>, Ref: #Ref<0.292262878.1007157249.121011>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.560 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7301.0>, Ref: #Ref<0.292262878.1007157249.121022>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
In previous set up we tried to swap peerbook from old version that was backed up
APshenkin commented
Finally after switching to docker images provided by you from https://quay.io/repository/team-helium/blockchain-node?tab=tags
BTW not clear for me why our images builded from source didn't work
Here is our Dockerfile
FROM erlang:22.3.2-alpine as builder
ARG CONFIG
RUN apk add --no-cache --update \
git tar build-base linux-headers autoconf automake libtool pkgconfig \
dbus-dev bzip2 bison flex gmp-dev cmake lz4 libsodium-dev openssl-dev \
sed curl cargo
# Install Rust toolchain
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
WORKDIR /usr/src/node
ENV CC=gcc CXX=g++ CFLAGS="-U__sun__" \
ERLANG_ROCKSDB_OPTS="-DWITH_BUNDLE_SNAPPY=ON -DWITH_BUNDLE_LZ4=ON" \
ERL_COMPILER_OPTIONS="[deterministic]" \
PATH="/root/.cargo/bin:$PATH" \
RUSTFLAGS="-C target-feature=-crt-static"
ARG TAG=1.1.66
# Add our code
RUN git clone https://github.com/helium/blockchain-node.git /usr/src/node && git checkout $TAG
# copy testnet config
COPY . /usr/src/node
RUN ./rebar3 as $CONFIG tar
RUN mkdir -p /opt/docker
RUN tar -zxvf _build/$CONFIG/rel/*/*.tar.gz -C /opt/docker
RUN mkdir -p /opt/docker/update
FROM erlang:22.3.2-alpine as runner
RUN apk add --no-cache --update ncurses dbus gmp libsodium gcc
RUN ulimit -n 64000
WORKDIR /opt/node
ENV COOKIE=node \
# Write files generated during startup to /tmp
RELX_OUT_FILE_PATH=/tmp \
# add miner to path, for easy interactions
PATH=$PATH:/opt/node/bin
COPY --from=builder /opt/docker /opt/node
ENTRYPOINT ["/opt/node/bin/blockchain_node"]
CMD ["foreground"]
Any thoughts?
Vagabond commented
We've found the bug is related to OTP22, use an OTP24 builder in your dockerfile
madninja commented
This issue belongs in the blockchain-node repo since that’s where the Dockerfile lives. It was addressed there already