BitcoinExchangeFH - Cryptocurrency exchange market data feed handler

BitcoinExchangeFH is a slim application to record the price depth and trades in various exchanges. You can set it up quickly and record the all the exchange data in a few minutes!

Users can

Streaming market data to a target application (via ZeroMQ)
Recording market data for backtesting and analysis.
Recording market data to a in-memory database and other applications can quickly access to it.
Customize the project for trading use.

MySQL

Kdb+

Supported exchanges

All exchanges supported by ccxt. Currently more than 130 exchanges are supported.

Websocket feeds of the following exchanges are supported by cryptofeed

Bitfinex
Coinbase
Poloniex
Gemini
HitBTC
Bitstamp
BitMEX
Kraken
Binance
EXX
Huobi
OKCoin
OKEx

If the exchange is not supported with websocket API feed, it will automatically fall into using its REST API feed.

Supported database/channel

RDMBS (e.g. sqlite, MySQL, PostgreSQL)
ZeroMQ
Kdb+ (Coming soon)

Getting started

pip install bitcoinexchangefh
bitcoinexchangefh --configuration example/configuration.yaml

Configuration

The configuration follows YAML syntax and contains two sections

subscriptions
handlers

Subscriptions

Subscription section specifies the exchange and instruments to subscribe.

The first key is the exchange name and then follows the exchange details,

instruments
number of depth (default is 5 if not specified)

For example,

subscription:
    Binance:
        instruments:
            - XRP/BTC
            - BCH/BTC
        depth
    Poloniex:
        instruments:
            - ETH/BTC
        depth: 10

Handlers

After receiving the order book or trade update, each handler is updated. For example, for SQL database handler, it is updated with the corresponding SQl statements.

For example,

handlers:
    sql: 
        connection: "sqlite://"

SQL handler

The following settings can be customized

Parameter	Description
connection	Database connection string required by SQLAlchemy
is_rotate	Boolean indicating whether to rotate to record the table.
rotate_frequency	String in format same as `strftime` and `strptime`

ZeroMQ handler

The feed handler acts as a publisher in ZeroMQ. To receive the feed, please follow ZeroMQ instructions to start a subscriber.

The following settings can be customized

Parameter	Description
connection	Connection format in ZeroMQ. For example, "tcp://127.0.0.1:3456"

Examples

You can first create a directory .data and run the command

$ bitcoinexchangefh --configuration example/configuration.yaml 
2020-08-07 23:33:32,110 INFO Loading runner
2020-08-07 23:33:32,110 INFO Creating handler sql
2020-08-07 23:33:32,205 INFO Loading handler SqlHandler
2020-08-07 23:33:32,710 INFO Loading exchange Binance
2020-08-07 23:33:34,062 INFO Loading exchange Bitmex
2020-08-07 23:33:40,203 INFO Start running the feed handler
2020-08-07 23:33:40,203 INFO Running handler sql
2020-08-07 23:33:40,210 INFO Running exchange Binance
2020-08-07 23:33:40,215 INFO Running SqlHandler
2020-08-07 23:33:40,219 INFO Running exchange Bitmex
2020-08-07 23:33:40,231 INFO Joining all the processes
2020-08-07 23:33:40,232 INFO Creating table binance_ethbtc_order
2020-08-07 23:33:40,263 INFO Created table binance_ethbtc_order
2020-08-07 23:33:40,265 INFO Creating table binance_xrpbtc_order
2020-08-07 23:33:40,294 INFO Created table binance_xrpbtc_order
2020-08-07 23:33:40,297 INFO Creating table bitmex_ethusd_order
2020-08-07 23:33:40,322 INFO Created table bitmex_ethusd_order

After a while, the data is streamed into the file .data/order_book.db. You can terminate the process and the database contains the following tables.

$ python -c "import sqlite3; print(sqlite3.connect('.data/order_book.db').execute('SELECT name from sqlite_master where type= \"table\"').fetchall())"
[('binance_ethbtc_order',), ('binance_xrpbtc_order',), ('bitmex_ethusd_order',)]

Inquiries

You can first look up to the page FAQ. For more inquiries, you can either leave it in issues or drop me an email. I will get you back as soon as possible.

Compatibility

The application is compatible with version higher or equal to python 3.0.

Contributions

Always welcome for any contribution. Please fork the project, make the changes, and submit the merge request. :)

For any questions and comment, please feel free to contact me through email (gavincyi at gmail)

Your comment will be a huge contribution to the project!

Continuity

If you are not satisified with python performance, you can contact me to discuss migrating the project into other languages, e.g. C++.

m-santa1718/BitcoinExchangeFH