coinbase/mesh-bitcoin

[bug] invalid memory address or nil pointer dereference when syncing

Closed this issue · 3 comments

shermanxiong commented

Steps to Reproduce:

  1. run make run-testnet-online
  2. let the node sync until around block 132800-132900
  3. see golang panic (below)

Logs from first time producing error - panic after processing block 132914:

2020-11-09T21:55:46.619Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000002d44c2249c29c1130d617b13beb5dd5261f1287e7c1ac3e90a51cdf9 height=132908 version=0x00000002 log2_work=57.187135 tx=724624 date='2013-11-10T20:52:01Z' progress=0.037070 cache=55.6MiB(401169txo)
2020-11-09T21:55:46.627Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000279266a5913eaf293c7cfefd5e8cb5e5f10b1f7ccddc362ea59d83a1 height=132909 version=0x00000002 log2_work=57.187135 tx=724625 date='2013-11-10T20:52:09Z' progress=0.037070 cache=55.6MiB(401170txo)
2020-11-09T21:55:46.637Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000003dda93b55aefbc85d6c4ee08a6151f05d51b84bb64386763164f37a1 height=132910 version=0x00000002 log2_work=57.187135 tx=724626 date='2013-11-10T20:52:13Z' progress=0.037070 cache=55.6MiB(401171txo)
2020-11-09T21:55:46.645Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000002e4da492f8ab8935ba8c73db05c9d79ab6db94fc844ccca13ffdce7a height=132911 version=0x00000002 log2_work=57.187135 tx=724627 date='2013-11-10T20:52:17Z' progress=0.037070 cache=55.6MiB(401172txo)
2020-11-09T21:55:46.651Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000003f55e3178660739e3ea48b277fc7176dcde5cd706813508c5dfb008a height=132912 version=0x00000002 log2_work=57.187135 tx=724628 date='2013-11-10T20:52:23Z' progress=0.037070 cache=55.6MiB(401173txo)
2020-11-09T21:55:46.657Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000002c720d115a6948ba146eeb4c7bbd698680b63bbf7818f489cea8d616 height=132913 version=0x00000002 log2_work=57.187136 tx=724629 date='2013-11-10T20:52:26Z' progress=0.037070 cache=55.6MiB(401174txo)
2020-11-09T21:55:46.664Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000001c8093b31d0a54a3aee3a5fefdd58ac535382b0cb084ca6222b84ff0 height=132914 version=0x00000002 log2_work=57.187136 tx=724630 date='2013-11-10T20:52:33Z' progress=0.037070 cache=55.6MiB(401175txo)
2020/11/09 21:55:46 Syncing 132711-132874
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xa1aa87]

goroutine 98 [running]:
github.com/coinbase/rosetta-sdk-go/storage.(*BlockStorage).AddBlock(0xc00009d740, 0xe92420, 0xc0000ccb80, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/storage/block_storage.go:545 +0xe7
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).BlockAdded(0xc000626100, 0xe92420, 0xc0000ccb80, 0x0, 0x0, 0x0)
	/app/src/indexer/indexer.go:320 +0xae
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).processBlock(0xc017cd23c0, 0xe92420, 0xc0000ccb80, 0xc0011a8f00, 0xc000083d01, 0xc00e87e110)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/syncer/syncer.go:318 +0x154
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).processBlocks(0xc017cd23c0, 0xe92420, 0xc0000ccb80, 0xc000bddd20, 0x2070a, 0xc000029e40, 0x0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/syncer/syncer.go:483 +0x12a
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).syncRange(0xc017cd23c0, 0xe92420, 0xc0000ccb80, 0x2070a, 0x2, 0x0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/syncer/syncer.go:628 +0x42e
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).Sync(0xc017cd23c0, 0xe92420, 0xc0000ccb80, 0x20667, 0xffffffffffffffff, 0x0, 0x0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/syncer/syncer.go:698 +0x3af
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).Sync(0xc000626100, 0xe92420, 0xc0000ccb80, 0x0, 0x0)
	/app/src/indexer/indexer.go:270 +0x2e5
main.startOnlineDependencies.func2(0x0, 0x0)
	/app/src/main.go:104 +0x3c
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000155b30, 0xc00061e300)
	/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:54 +0x66

Logs from second time producing same error but at a different block height - 132809

2020-11-09T22:45:10.990Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000001249293c57bea9ad24483dab5f9a8537f372a9ab7b5ed985006095e3 height=132803 version=0x00000002 log2_work=57.187119 tx=724510 date='2013-11-10T20:44:26Z' progress=0.037064 cache=55.6MiB(401065txo)
2020-11-09T22:45:11.001Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000001b96a6567aa04e7641dcd937957aa8c584f141e71cbb25a71a724179 height=132804 version=0x00000002 log2_work=57.187119 tx=724511 date='2013-11-10T20:44:28Z' progress=0.037064 cache=55.6MiB(401066txo)
2020-11-09T22:45:11.015Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000042681c96a953abf2d43a0bb4fa445e603c0e6005b103377ec351fd3 height=132805 version=0x00000002 log2_work=57.187119 tx=724512 date='2013-11-10T20:44:34Z' progress=0.037064 cache=55.6MiB(401067txo)
2020/11/09 22:45:11 Syncing 132711-132788
2020-11-09T22:45:11.032Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000126dc0b2a376210d72dbc785c600f2c82ef5e23f2d133e79ae094661 height=132806 version=0x00000002 log2_work=57.187119 tx=724514 date='2013-11-10T20:44:41Z' progress=0.037064 cache=55.6MiB(401066txo)
2020-11-09T22:45:11.046Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000000b62a79ff975c7f42918eefc9e30412f004ac74e50fbd336b617d91d height=132807 version=0x00000002 log2_work=57.187119 tx=724515 date='2013-11-10T20:44:47Z' progress=0.037064 cache=55.6MiB(401067txo)
2020-11-09T22:45:11.061Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000284b13d618fad2cb5b51356b88b3e45cf9f1d49788810f89d9c5ea84 height=132808 version=0x00000002 log2_work=57.18712 tx=724516 date='2013-11-10T20:44:54Z' progress=0.037064 cache=55.6MiB(401068txo)
2020-11-09T22:45:11.075Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000112dd164508c156037048123be1c9c6479dce4b001cde45ee69f9175 height=132809 version=0x00000002 log2_work=57.18712 tx=724517 date='2013-11-10T20:45:03Z' progress=0.037064 cache=55.6MiB(401069txo)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xa1aa87]

goroutine 86 [running]:
github.com/coinbase/rosetta-sdk-go/storage.(*BlockStorage).AddBlock(0xc0248b9770, 0xe92420, 0xc0000ccb80, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/storage/block_storage.go:545 +0xe7
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).BlockAdded(0xc00055c400, 0xe92420, 0xc0000ccb80, 0x0, 0x0, 0x0)
	/app/src/indexer/indexer.go:320 +0xae
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).processBlock(0xc0002386c0, 0xe92420, 0xc0000ccb80, 0xc017ca7540, 0xc000682b01, 0xc027810ef0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/syncer/syncer.go:318 +0x154
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).processBlocks(0xc0002386c0, 0xe92420, 0xc0000ccb80, 0xc000175d20, 0x206b4, 0x2a, 0x0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/syncer/syncer.go:483 +0x12a
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).syncRange(0xc0002386c0, 0xe92420, 0xc0000ccb80, 0x206b4, 0x2, 0x0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/syncer/syncer.go:628 +0x42e
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).Sync(0xc0002386c0, 0xe92420, 0xc0000ccb80, 0x20667, 0xffffffffffffffff, 0x0, 0x0)
	/go/pkg/mod/github.com/coinbase/rosetta-sdk-go@v0.5.8/syncer/syncer.go:698 +0x3af
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).Sync(0xc00055c400, 0xe92420, 0xc0000ccb80, 0x0, 0x0)
	/app/src/indexer/indexer.go:270 +0x2e5
main.startOnlineDependencies.func2(0x0, 0x0)
	/app/src/main.go:104 +0x3c
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000155b30, 0xc00000e9a0)
	/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.0.0-20200625203802-6e8e738ad208/errgroup/errgroup.go:54 +0x66
patrick-ogrady commented

I'm working on trying to reproduce this now! I've unsuccessfully attempted to do so a few times so have reason to believe this is some sort of race condition.

shermanxiong commented

Turns out it was as data corruption issue. What happened was i initially ran make run-mainnet-online, then killed it and then started running make run-testnet-online without cleaning the bitcoin-data directory first.

Solution: remove the bitcoin-data directory (you may need to use sudo) if you have run one network and want to switch to running another.

  • 👍1
patrick-ogrady commented

May make sense to use a different directory name for each network. I'll add a ticket to update the README and Makefile to advise this.