[Polygon Mainnet] Snapshot download issue
Closed this issue · 3 comments
System information
Erigon version: 2.59.3
OS & Version: Amazon Linux 2023
Commit hash: 088fd8e
Erigon Command (with flags/config): erigon --nat=extip:<ip> --identity=<identity> --txpool.disable=true --ethash.dagdir=/home/polygon/data/erigon-ethash --bor.heimdall=http://<heimdall url>. --txpool.nolocals=true --datadir=/home/polygon/data/bor --http.addr=0.0.0.0 --http.corsdomain=* --torrent.download.slots=10000 --http.api=admin,eth,net,web3,debug,engine,erigon,bor --ws --gpo.percentile=65 --rpc.batch.concurrency=100 --http.vhosts=* --rpc.batch.limit=3000 --torrent.download.rate=3000mb --chain=bor-mainnet
Consensus Layer: N/A
Consensus Layer Command (with flags/config): N/A
Chain/Network: bor-mainnet
Expected behaviour
To download all the snapshots and sync from scratch.
Actual behaviour
It gets stuck downloading the snapshot.
Steps to reproduce the behaviour
I've simply started the node (tried at least 4 times from scratch) and it always gets stuck around ~95%. I'm not using any public snapshot (like from https://snapshots.polygon.technology/) to start my node. I'm simply starting it in an empty disk.
Not sure if I'm missing any magical flag, this is my first try with erigon for polygon. I need a full history archival node, this is why I was willing to sync from scratch without using any pruned snapshot. On the 4 times I tried, I got the same result, before each try I was simply running rm- rf ./* and starting erigon again with the flags mentioned above.
There are no firewall rules blocking egress requests. Any recommendations to have this working?
Backtrace
[INFO] [05-03|23:09:35.406] logging to file system log dir=/home/polygon/data/bor/logs file prefix=erigon log level=info json=false
[INFO] [05-03|23:09:35.406] Build info git_branch=HEAD git_tag=v2.59.3 git_commit=088fd8ef69389a72da6faa0fc7903a4ba5726911
[INFO] [05-03|23:09:35.406] Starting Erigon on Bor Mainnet...
[INFO] [05-03|23:09:35.408] Maximum peer count ETH=100 total=100
[INFO] [05-03|23:09:35.408] starting HTTP APIs port=8545 APIs=admin,eth,net,web3,debug,engine,erigon,bor
[INFO] [05-03|23:09:35.409] torrent verbosity level=WRN
[INFO] [05-03|23:09:35.409] [torrent] Public IP ip=<>
[INFO] [05-03|23:09:35.410] Set global gas cap cap=50000000
[INFO] [05-03|23:09:35.413] [Downloader] Running with ipv6-enabled=true ipv4-enabled=true download.rate=3000mb upload.rate=4mb
[INFO] [05-03|23:09:35.413] Opening Database label=chaindata path=/home/polygon/data/bor/chaindata
[INFO] [05-03|23:09:35.414] [db] open lable=chaindata sizeLimit=12TB pageSize=8192
[INFO] [05-03|23:09:35.417] Re-Opening DB in exclusive mode to apply migrations
[INFO] [05-03|23:09:35.418] [db] open lable=chaindata sizeLimit=12TB pageSize=8192
[INFO] [05-03|23:09:35.419] Apply migration name=db_schema_version5
[INFO] [05-03|23:09:35.420] Applied migration name=db_schema_version5
[INFO] [05-03|23:09:35.420] Apply migration name=txs_begin_end
[INFO] [05-03|23:09:35.422] Applied migration name=txs_begin_end
[INFO] [05-03|23:09:35.422] Apply migration name=txs_v3
[INFO] [05-03|23:09:35.422] Applied migration name=txs_v3
[INFO] [05-03|23:09:35.422] Apply migration name=prohibit_new_downloads_lock
[INFO] [05-03|23:09:35.423] Applied migration name=prohibit_new_downloads_lock
[INFO] [05-03|23:09:35.424] Updated DB schema to version=6.1.0
[INFO] [05-03|23:09:35.425] [db] open lable=chaindata sizeLimit=12TB pageSize=8192
[INFO] [05-03|23:09:35.433] Writing custom genesis block hash=0xa9c28ce2141b56c474f1dc504bee9b01eb1bd7d1a507580d5519d4437a97de1b
[INFO] [05-03|23:09:35.435] Initialised chain configuration config="{ChainID: 137, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: 23850000, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Shanghai: <nil>, Cancun: <nil>, Prague: <nil>, Osaka: <nil>, Engine: bor, NoPruneContracts: map[]}" genesis=0xa9c28ce2141b56c474f1dc504bee9b01eb1bd7d1a507580d5519d4437a97de1b
[INFO] [05-03|23:09:35.444] Initialising Ethereum protocol network=137
[INFO] [05-03|23:09:35.444] Opening Database label=bor path=/home/polygon/data/bor/bor
[INFO] [05-03|23:09:35.451] Starting private RPC server on=127.0.0.1:9090
[INFO] [05-03|23:09:35.451] new subscription to logs established
[INFO] [05-03|23:09:35.451] rpc filters: subscribing to Erigon events
[INFO] [05-03|23:09:35.453] new subscription to newHeaders established
[INFO] [05-03|23:09:35.454] JsonRpc endpoint opened ws=true ws.compression=true grpc=false websocket.url=[::]:8546 http.url=[::]:8545
[INFO] [05-03|23:09:35.463] [1/15 Snapshots] Requesting downloads
[INFO] [05-03|23:09:35.465] Started P2P networking version=67 self=enode://<enode>@<ip>:30304 name=<name>/v2.59.3-088fd8ef/linux-arm64/go1.22.0
[INFO] [05-03|23:09:35.466] Started P2P networking version=68 self=enode://<enode>@<ip>:30303 name=<name>/v2.59.3-088fd8ef/linux-arm64/go1.22.0
[INFO] [05-03|23:09:35.879] [snapshots] no metadata yet files=562 list=v1-005000-005500-transactions.seg,v1-044500-045000-bodies.seg,v1-052800-052900-bodies.seg,v1-012000-012500-headers.seg,v1-036500-037000-headers.seg,...
[INFO] [05-03|23:09:55.502] [snapshots] no progress yet files=151 list=v1-025500-026000-transactions.seg,v1-045500-046000-borspans.seg,v1-029000-029500-borspans.seg,v1-042000-042500-transactions.seg,v1-017500-018000-borspans.seg,...
[INFO] [05-03|23:09:55.879] [1/15 Snapshots] download progress="0.04% 416.8MB/905.3GB" time-left=13hrs:20m total-time=20s download=19.3MB/s upload=0B/s peers=4 files=562 metadata=562/562 connections=9 alloc=518.0MB sys=1.0GB
[INFO] [05-03|23:10:15.499] [snapshots] no progress yet files=143 list=v1-034500-035000-bodies.seg,v1-051500-051600-headers.seg,v1-042500-043000-borspans.seg,v1-032500-033000-borspans.seg,v1-003500-004000-borevents.seg,...
[INFO] [05-03|23:10:15.879] [1/15 Snapshots] download progress="0.12% 1.1GB/905.3GB" time-left=7hrs:8m total-time=40s download=36.0MB/s upload=0B/s peers=5 files=562 metadata=562/562 connections=40 alloc=647.7MB sys=1.1GB
[INFO] [05-03|23:10:35.529] [snapshots] no progress yet
...
...
[INFO] [05-04|12:06:55.880] [1/15 Snapshots] download progress="95.40% 863.7GB/905.3GB" time-left=999hrs:99m total-time=12h57m20s download=0B/s upload=0B/s peers=56 files=562 metadata=562/562 connections=95 alloc=865.0MB sys=2.1GB
[INFO] [05-04|12:07:15.532] [snapshots] no progress yet files=6 list=v1-051000-051100-transactions.seg,v1-043500-044000-transactions.seg,v1-052400-052500-transactions.seg,v1-051700-051800-headers.seg,v1-052700-052800-bodies.seg,...
[INFO] [05-04|12:07:15.880] [1/15 Snapshots] download progress="95.40% 863.7GB/905.3GB" time-left=999hrs:99m total-time=12h57m40s download=0B/s upload=0B/s peers=56 files=562 metadata=562/562 connections=95 alloc=950.0MB sys=2.1GB
[INFO] [05-04|12:07:35.529] [snapshots] no progress yet files=6 list=v1-051000-051100-transactions.seg,v1-043500-044000-transactions.seg,v1-052400-052500-transactions.seg,v1-051700-051800-headers.seg,v1-052700-052800-bodies.seg,...
[INFO] [05-04|12:07:35.880] [1/15 Snapshots] download progress="95.40% 863.7GB/905.3GB" time-left=999hrs:99m total-time=12h58m0s download=0B/s upload=0B/s peers=57 files=562 metadata=562/562 connections=95 alloc=1.0GB sys=2.1GB
[INFO] [05-04|12:07:55.531] [snapshots] no progress yet files=6 list=v1-053100-053200-headers.seg,v1-051000-051100-transactions.seg,v1-043500-044000-transactions.seg,v1-052400-052500-transactions.seg,v1-051700-051800-headers.seg,...
[INFO] [05-04|12:07:55.880] [1/15 Snapshots] download progress="95.40% 863.7GB/905.3GB" time-left=999hrs:99m total-time=12h58m20s download=0B/s upload=0B/s peers=58 files=562 metadata=562/562 connections=95 alloc=1.1GB sys=2.1GB
And then it gets stuck repeating these same lines over and over again, never downloading these last files.
I've decided to try a 5th time, and this time it managed to progress farther, but still got stuck at some point.
[INFO] [05-04|13:30:53.087] [1/15 Snapshots] download progress="99.90% 906.7GB/905.3GB" time-left=999hrs:99m total-time=48m20s download=0B/s upload=0B/s peers=19 files=562 metadata=562/562 connections=36 alloc=626.1MB sys=2.5GB
[polygon@bor-erigon-0 data]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/nvme1n1 10T 906G 9.1T 9% /home/polygon/data
...[polygon@bor-erigon-0 data]$ du -h bor/snapshots/
4.0K bor/snapshots/accessor
4.0K bor/snapshots/idx
4.0K bor/snapshots/domain
4.0K bor/snapshots/history
906G bor/snapshots/Try start once with —downloader.verify flag
Before using the -downloader.verify I decided to give one last try. And in this one last try I also downgraded go from 1.22.0 to 1.21.9. I believe this was just a coincidence, but this time it managed to finish downloading the whole thing and progressed to the [1/15 Snapshots] Indexing step. I'm closing this one, coincidence or not, the 9th try was a charm.