lioneltayyd/compile-ticker-data

Automate the process of data collecting, preprocessing, and dashboard update for market analysis with Luigi pipeline tool.

Compile Ticker Data

Project Purpose

To compile ticker data from internet resources to explore and analyse the ETF and equity trades.

Main Tools

Tools	Description
luigi	For building data pipeline and dashboard.
pandas	For data processing.
pipenv	For managing dependencies.

Data Architecture Overview

This project focuses on the Streamlit Dashboard section. It's associated with Market Trades Dashboard Repo (Streamlit Dashboard section).

Folder Structure

File / Folder Name	Description
autovisualise_data	For keeping custom python modules related to data visualisation.
config	For configuration. It encompasses 3 files. config_logger is for logger, config_dashboard for dashboard, config_naming for namings, config for other general configuration.
docs	For storing files, data, and documents.
logs	For storing the log info.
sh	For running bash script on Mac.
pipfile	For setting up the virtual environment and tracking all the installed dependencies.

Project Setup

1. Pull the project from the repo.

2. Install pyenv to manage the python version (if needed) and pipenv for dependencies.

3. Run this to install dependencies from the Pipfile.lock file.

   pipenv shell;
   pipenv sync; 

4. To reinstall the entire dependencies, update the version within the Pipfile file if needed, then run this. This will automatically create / update the Pipfile.lock file for you.

   pipenv install;

5. Try import specific library and check the version to see if it's installed.

Code Running Guide

1. Run the streamlit dashboard app locally.

   streamlit run run_streamlit.py

2. Run the docker compose to start the app locally.

   sh sh/run_docker_compose.sh up

   To stop the app from running, do this.

   sh sh/run_docker_compose.sh down 

Deployment Guide

Updating soon...

Debugging & Testing

1. To debug specific functions or code, use Jupytext Percent Format. Simple include the following at the bottom of the run_pipeline.py file and write your code to test specific function(s). This relies on ipykernel module so ensure that it is installed via pipenv.