/Backtesting-Infrastructure-CryptoTrading

A scalable, Dockerized microservices infrastructure for backtesting crypto trading strategies. It automates data collection, backtesting, and strategy management, leveraging Kafka for real-time streaming, FastAPI for efficient interaction, WebSockets for live updates and ReactJS for the frontend.

Primary LanguageJupyter Notebook

Crypto Trading Engineering: Scalable Backtesting Infrastructure

Build Status

Overview

This project is a robust trading and backtesting infrastructure designed to support the evaluation of various trading strategies on financial instruments like stocks and cryptocurrencies. It leverages a microservices architecture with Docker for flexibility and scalability, using tools like Airflow, Kafka, FastAPI, and PostgreSQL.

Key Features

  • Automated Data Pipeline: Fetches historical market data using yfinance, processes it, and stores it in a PostgreSQL database.
  • Backtesting Engine: Evaluates trading strategies against historical or real-time market data.
  • Strategy Management: Allows the creation and storage of custom trading strategies.
  • Kimball Schema: Employs a Kimball data warehouse schema for efficient storage and analysis of data.
  • Microservice Architecture: Utilizes Docker containers for modularity and scalability.
  • Kafka Integration: Employs Kafka for real-time data streaming and message passing.
  • FastAPI Backend: Provides a modern REST API for interacting with the system.
  • WebSocket Support: Enables real-time updates and data streaming to the frontend.
  • Scalability: Designed to handle increasing volumes of data and backtest scenarios.

Project Structure

  • airflow/: Contains Airflow DAGs for orchestrating data pipelines and backtests.
  • api/: Houses the FastAPI backend service.
  • database_service/: Contains the database service logic for data access and storage.
  • kafka_consumer/: Holds the Kafka consumer service for data ingestion.
  • models/: Defines the SQLAlchemy models for the database schema.
  • shared/: Stores shared modules (e.g., utility functions, configurations).
  • src/: Contains source code for data fetching, backtesting logic, and strategies.
  • tests/: Includes unit and integration tests for the project.

Getting Started

  1. Prerequisites:
    • Docker and Docker Compose
    • Python 3.9+ (with required dependencies: see requirements.txt)

Installation

  • Fork the repository
  • Clone the repository you just forked using git clone

In your terminal:

cd Backtesting-Infrastructure-CryptoTrading
pip install -r requirements.txt

Usage

In your terminal run:

make up

This will start the services in compose.yaml file. You can check the status of the services by running:

docker ps

To check the logs of a service, run:

docker compose logs -f <service_name>

Where the service_name is the name of the service found in the compose.yaml file.

  • Example of checking the logs of the api service:

    docker compose logs -f api

Accessing Services

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.