eth-indexer

eth-indexer is an Ethereum blockchain indexer project to crawl blocks, transactions & state difference per block/address into MySQL database.

Getting Started

There are 3 main components in the project:

geth: modified geth to get state difference per block/address
idx-database: MySQL to store all indexed data
indexer: indexer to crawl from geth then push to database

Prerequisites

docker
docker-compose

Before Building

Before building, please make sure environment variables MYSQL_DATA_PATH and GETH_DATA_PATH are setup properly, which are used to mount local data folder to MySQL and Geth containers for data persistence. One way to set this up is to have a .env file in the same folder of the docker-compose.yml

Example .env file:

MYSQL_DATA_PATH=~/indexer-data/mysql
GETH_DATA_PATH=~/indexer-data/geth

Configs and Flags

eth_indexer supports two kinds of input:

static config YAML files
dynamic flags through command line

You can either define your configs/config.yml or pass flags (e.g., indexer --eth.port 1234) from command line to start eth_indexer. If you use both settings, eth_indexer will load configs/config.yaml as default and overwrite the corresponding values with specified flags from command line.

Build

$ git clone git@github.com:getamis/eth-indexer.git
$ cd eth-indexer
$ # Set MYSQL_DATA_PATH and GETH_DATA_PATH environment variables
$ docker-compose build

Usage

We use docker-compose for testing and developing. MYSQL_DATA_PATH & GETH_DATA_PATH environment variables are necessary, create them out of eth-indexer directory to store database and geth data.

first time to run indexer you need to create the database schema

$ mkdir -p ~/indexer-data/mysql ~/indexer-data/geth
# Create database sechema
MYSQL_DATA_PATH="$HOME/indexer-data/mysql" docker-compose up idx-database idx-migration
# press Ctrl + C when see `eth-indexer_idx-migration_1 exited with code 0`

then use docker-compose up with environment variables to start indexer:

$ MYSQL_DATA_PATH="$HOME/indexer-data/mysql" GETH_DATA_PATH="$HOME/indexer-data/geth" docker-compose up

wait few minutes, then you can see indexing messages from indexer:

Inserted TD for block                    number=0       TD=17179869184 hash=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3

Example

Once there are some data in MySQL, you can query specific data from it, e.g., you can get data from block_headers and transactions table. Balance is slightly different, and you can take a look at example folder to see how to query them.

package main

import (
	"context"
	"fmt"

	"github.com/ethereum/go-ethereum/common"
	"github.com/getamis/eth-indexer/model"
	"github.com/getamis/eth-indexer/store/account"
	"github.com/getamis/eth-indexer/store/sqldb"
	"github.com/getamis/sirius/database"
	"github.com/getamis/sirius/database/mysql"
)

func main() {
	db, _ := sqldb.New("mysql",
		database.DriverOption(
			mysql.Database("ethdb"),
			mysql.Connector(mysql.DefaultProtocol, "127.0.0.1", "3306"),
			mysql.UserInfo("root", "my-secret-pw"),
		),
	)
	addr := common.HexToAddress("0x756f45e3fa69347a9a973a725e3c98bc4db0b5a0")
	store := account.NewWithDB(db)

	account, err := store.FindAccount(context.Background(), model.ETHAddress, addr)
	if err != nil {
		fmt.Printf("Failed to find account: %v\n", err)
	} else {
		fmt.Printf("Find account, block_number: %v, balance: %v, \n", account.Balance, account.BlockNumber)
	}
}

ERC20 is similar, and you can see the test case for ERC20 to know how to use it.

Contributing

There are several ways to contribute to this project:

Find bug: create an issue in our Github issue tracker.
Fix a bug: check our issue tracker, leave comments and send a pull request to us to fix a bug.
Make new feature: leave your idea in the issue tracker and discuss with us then send a pull request!

License

This project is licensed under the LGPL 3 - see the LICENSE file for details

tomi-lui/eth-indexer