Squirrel is a program that parses and persists NEO blockchain data into MySQL database, including UTXO transfers, NEP5 transfers, address balances and other statistical data.
- Fast blockchain data sync.
- Supports multi-fullnode as data sources to avoid SPOF.
- Independent sync targets in different task goroutines, easy to modify, stop or restart.
- Mail alert if unexpected panic happened(Optional).
- All are sql transactions, safe to
Ctrl+C
at any time for any reason.
Go 1.12+ is recommended to build and run this program. For Go 1.11.*, please make sure the environment variable
GO111MODULE
is set toon
.
- Clone this repo to your local directory:
git clone https://github.com/NeoNextClub/squirrel.git
- Create database:
Execute all SQL statements in ./sqls/create_table.sql
to create database and tables.
- Copy
config.sample.json
asconfig.json
and fill the configurations:
cp ./config/config.sample.json ./config/config.json
vim config/config.json
- Build source code and run:
# For go 1.13 and above, the following go proxy is recommended:
# go env -w GOPROXY=https://goproxy.cn,direct
# Visit https://goproxy.cn for detailed info.
# Install dependencies and build binary
go get -v -t ./... && go build
# Run squirrel, you can attach -mail flag to enable mail alert.
./squirrel
Here is the space size used on different networks(for reference) under MySQL 5.7.
Network | Block Height | Size |
---|---|---|
Mainnet | 5,320,000 | ~156G |
TestNet | 4,070,000 | ~27G |
Currently there are 3 individual tasks except the main block data sync task:
- NEP5 Transfers Sync Task.
- UTXO Transfers Sync Task.
- gas Daily Balance sync Task.
If you want to add extra columns or make some changes to the existing tables that being used by these tasks, then you may want to restart these tasks from beginning, to clear these related tables/data, run the following sqls:
💡 Please make sure the
squirrel
program is stopped before executing these sqls.
NEP5 Transfers Sync Task
DELETE FROM `addr_asset` WHERE LENGTH(`asset_id`) = 40;
DELETE FROM `addr_tx` WHERE `asset_type` = 'nep5';
UPDATE `address` SET `trans_nep5` = 0 WHERE 1 = 1;
UPDATE `counter` SET
`last_tx_pk_for_nep5` = 0,
`app_log_idx` = -1
WHERE `id` = 1;
TRUNCATE TABLE `nep5`;
TRUNCATE TABLE `nep5_reg_info`;
TRUNCATE TABLE `nep5_tx`;
TRUNCATE TABLE `nep5_migrate`;
DELETE FROM `address` WHERE `trans_asset` = 0 AND `trans_nep5` = 0;
UPDATE `counter` SET `nep5_tx_pk_for_addr_tx` = 0 WHERE `id` = 1;
UTXO Transfers Sync Task
TRUNCATE TABLE `utxo`;
DELETE FROM `addr_asset` WHERE LENGTH(`asset_id`) = 66;
DELETE FROM `addr_tx` WHERE `asset_type`='asset';
UPDATE `counter` SET
`last_tx_pk` = 0,
`cnt_tx_reg` = 0,
`cnt_tx_miner` = 0,
`cnt_tx_issue` = 0,
`cnt_tx_invocation` = 0,
`cnt_tx_contract` = 0,
`cnt_tx_claim` = 0,
`cnt_tx_publish` = 0,
`cnt_tx_enrollment` = 0
WHERE `id` = 1;
UPDATE `asset` SET `addresses` = 0, `available` = 0, `transactions` = 0;
UPDATE `address` SET `trans_asset` = 0;
TRUNCATE TABLE `asset_tx`;
UPDATE `counter` SET `last_asset_tx_pk` = 0 WHERE `id` = 1;
GAS Daily Balance Sync Task
TRUNCATE TABLE `addr_gas_balance_a`;
TRUNCATE TABLE `addr_gas_balance_b`;
TRUNCATE TABLE `addr_gas_balance_c`;
TRUNCATE TABLE `addr_gas_balance_d`;
TRUNCATE TABLE `addr_gas_balance_e`;
TRUNCATE TABLE `addr_gas_balance_f`;
TRUNCATE TABLE `addr_gas_balance_g`;
TRUNCATE TABLE `addr_gas_balance_h`;
TRUNCATE TABLE `addr_gas_balance_i`;
TRUNCATE TABLE `addr_gas_balance_j`;
TRUNCATE TABLE `addr_gas_balance_k`;
TRUNCATE TABLE `addr_gas_balance_l`;
TRUNCATE TABLE `addr_gas_balance_m`;
TRUNCATE TABLE `addr_gas_balance_n`;
TRUNCATE TABLE `addr_gas_balance_o`;
TRUNCATE TABLE `addr_gas_balance_p`;
TRUNCATE TABLE `addr_gas_balance_q`;
TRUNCATE TABLE `addr_gas_balance_r`;
TRUNCATE TABLE `addr_gas_balance_s`;
TRUNCATE TABLE `addr_gas_balance_t`;
TRUNCATE TABLE `addr_gas_balance_u`;
TRUNCATE TABLE `addr_gas_balance_v`;
TRUNCATE TABLE `addr_gas_balance_w`;
TRUNCATE TABLE `addr_gas_balance_x`;
TRUNCATE TABLE `addr_gas_balance_y`;
TRUNCATE TABLE `addr_gas_balance_z`;
TRUNCATE TABLE `addr_gas_balance_0`;
TRUNCATE TABLE `addr_gas_balance_1`;
TRUNCATE TABLE `addr_gas_balance_2`;
TRUNCATE TABLE `addr_gas_balance_3`;
TRUNCATE TABLE `addr_gas_balance_4`;
TRUNCATE TABLE `addr_gas_balance_5`;
TRUNCATE TABLE `addr_gas_balance_6`;
TRUNCATE TABLE `addr_gas_balance_7`;
TRUNCATE TABLE `addr_gas_balance_8`;
TRUNCATE TABLE `addr_gas_balance_9`;
UPDATE `counter` SET `last_tx_pk_gas_balance` = 0 WHERE `id` = 1;