AragornJIA/bsc-etl

Binance Smart Chain (BSC) ETL

BSC ETL lets you convert blockchain data into convenient formats like CSVs and relational databases. Based around and uses the (ethereum-etl) as a library.

Full documentation available here.

Quickstart

Install BSC ETL:

pip3 install bsc-etl

Export blocks and transactions (Schema, Reference):

> bscetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://bsc-dataseed1.binance.org/v3/7aef3f0cd1f64408b163814b22cc643c

Export BEP20 transfers (Schema, Reference):

> bscetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/bsc/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> bscetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/bsc/parity.ipc --output traces.csv

---

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install bsc-etl[streaming]
> bscetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://bsc-dataseed1.binance.org/v3/7aef3f0cd1f64408b163814b22cc643c

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 bscetl.py

Useful Links

• Schema
• Command Reference
• Documentation
• Public Datasets in BigQuery
• Exporting the Blockchain
• Querying in Amazon Athena
• Querying in Google BigQuery
• Querying in Kaggle
• Airflow DAGs
• Postgres ETL

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> export PROVIDER_URL=<your_porvider_uri>
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

1. Install Docker: https://docs.docker.com/install/

2. Build a docker image

    > docker build -t bsc-etl:latest .
    > docker image ls
   

3. Run a container out of the image

    > docker run -v $HOME/output:/bscetl-etl/output bscetl-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://bsc-dataseed1.binance.org
    > docker run -v $HOME/output:/bscetl-etl/output bscetl-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://bsc-dataseed1.binance.org
   

4. Run streaming to console or Pub/Sub

    > docker build -t bsc-etl:latest -f Dockerfile .
    > echo "Stream to console"
    > docker run bscetl-etl:latest stream --start-block 500000 --log-file log.txt
    > echo "Stream to Pub/Sub"
    > docker run -v /path_to_credentials_file/:/bsc-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/bsc-etl/credentials_file.json bscetl-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_bsc
   

Projects using BSC ETL

** Add Some! **