Indexer got stuck in reverting status
Closed this issue · 9 comments
Hi,
Somehow our mainnet tzkt got stuck in reverting status and become out of sync...
Once we restart the indexer, it is not recovering also
we have only one rolling node to sync with
after rebooting seems it is back...I would assume tzkt is kinda self healing, please feel free to close it if that is the case, sorry for the inconvience
Great! Yeah, reverting is a more complicated process and in some specific cases may take longer time. So, basically, if there are no errors in the logs, then the indexer is not stuck but likely waiting for the DB, executing some slow query.
I also saw your message about debug log level. Have you managed to enable it? Setting Logging__LogLevel__Default actually should work. Note, those are double underscores.
Hi @Groxan yeah, setting log level via env var actually works, I think my concern was that tzkt becomes out of sync when the above symptoms happens. I will close it for now
Hi @Groxan unfortunately I am seeing this issue again, I bumped db timeout from 600 to 3000 and still hitting this error
like you said if there are no errors in the logs, but I do see that sync is stucked with fatal log level, any pointer would be appreciated...
The record in the logs fail: Invalid head. Reverting... means that there was chain reorg, and the latest block is no longer valid and will be reverted. Then you see debug: Revert block - this means that the indexer is in process of reverting the block. So, as you can see, it's not stuck with fatal.
Try to check Postgres' logs to see what happens there and on which SQL query the DB is stuck.
Thanks @Groxan much appreciate it !