Indexer is extremely slow
Closed this issue · 9 comments
I'm running rosetta-bitcoin on a 8vCPU 64GB vm with SSD storage backing the data. The bitcoin node is already synced till the tip however the indexer is too slow in catching up.
The indexer starts quite fast in the beginning but it slows down around 300k block and then syncs one block every 1-0.5s which will take forever to catchup with tip.
Any idea why this could be happening?
Attaching a screenshot of logs (You can see badger taking up minutes for compaction and also each block time diff is about a second)
Thanks for sharing this screenshot, @HelloKashif! Another user, @sidhujag, actually started a thread about the same topic. Encourage you to chime in there: syscoin#1 (comment)
In our blockchain there are offending blocks with lots of operations, this seems to also be happening in Bitcoin as 20k operations will start to bog down balance updates but I did do some optimizations and will PR tagging this issue for review.
I'm now trying to just throw more compute and RAM at it. Trying with a 32CPU/32GB RAM machine and see if it syncs better.
@HelloKashif jfyi in 80k operations it takes roughly 5 hours to validate on 16gb ram (100gb swap) 8CPU... the fix takes it down to 5-10 seconds and likely 1-2 seconds with your configuration. I think adding more memory is ideal you will still need swap a bit just in case.
@sidhujag sounds good. Looking forward for that PR to merge so I can try it.
Here you go! https://github.com/coinbase/rosetta-bitcoin/releases/tag/v0.0.9
Awesome awesome 🥳 . I'm testing this now and its so much faster than before.
For the first time I'm seeing that indexer is almost catching up to the syncing tip.
@patrick-ogrady Thanks a lot for all the awesome effort here.
Nice, seems like the indexer is almost as fast as the node behind it.
For syscoin its about the same. I see alot of "pausing" then syncing almost like sychronous waits. But also because our version uses the performance settings for table/value logs the memory is through the roof like 30+ gigs. So may be thrashing causing that and we need atleast 32GB ram or more likely. I haven't tried with default because we've seen it get stuck but if you think the badget tx commit size's are smaller @patrick-ogrady due to optimization I can give it a try.
I haven't tried with default because we've seen it get stuck but if you think the badget tx commit size's are smaller @patrick-ogrady due to optimization I can give it a try.
The pre-store optimization definitely made the tx commit size smaller (think of it like breaking it apart into 2 txs...1 for block data, 1 for balance changes and coin updates). Could definitely work if you try but can't really make a prediction without digging in a little deeper.