Bitcoin Price Prediction Through Twitter Sentiment and Data Volume

This is the official repository for the the Bitcoin Price Prediction Through Twitter Sentiment and Data Volume Project

Author: Jacques Vella Critien Supervisors: Dr Joshua Ellul, Prof Albert Gatt

Structure

project
│   README.md  
│
└───data_generators_cleaners
│   │   BTCprices_cleaner.ipynb
│   │   data_grouper.ipynb
│   │   crypto_prices_getter_yahoo.ipynb
│   │   data_lag_creator.ipynb
│   │   english_tweets_extractor.ipynb
│   │   polarity_adder.ipynb
│   │   tweet_preprocessor.py
│   │   tweets_cleaner_finaliser.ipynb
│   │   
│   └───with_sentiment
│       │   
│       └───VADER
│            │   tweets.csv
│            └───cleaned
│                │   tweets.csv
│   
└───datasets
│   │
│   └───general
│   │   │   BTCDATAwithdate.csv
│   │   │   BTCTWEETS_english_no_duplicates.csv
│   │   │   preprocessed_tweets.csv
│   │   │   tweets_cleaned.csv
│   │   │   BTCDATA.csv
│   │   │   BTCTWEETS.csv
│   │
│   └───tweets_prices
│   │   │
│   │   └───vader
│   │       │   final_days_lag_days_1.csv
│   │       │   final_days_lag_days_3.csv
│   │       │   final_days_lag_days_7.csv
│   │       │   final_days_lag_hours_1.csv
│   │       │   final_days_lag_hours_3.csv
│   │       │   final_days_lag_hours_7.csv
│   │       │   
│   │       └───cleaned
│   │           │   final_days_lag_days_1.csv
│   │           │   final_days_lag_days_3.csv
│   │           │   final_days_lag_days_7.csv
│   │           │   final_days_lag_hours_1.csv
│   │           │   final_days_lag_hours_3.csv
│   │           │   final_days_lag_hours_7.csv
│   │
│   └───tweets_prices_volumes_sentiment
│       │
│       └───vader
│           │
│           └───day_datasets
│               │   final_days_lag_days_1.csv
│               │   final_days_lag_days_3.csv
│               │   final_days_lag_days_7.csv
│               │   final_days_lag_hours_1.csv
│               │   final_days_lag_hours_3.csv
│               │   final_days_lag_hours_7.csv
│               │   
│               └───cleaned
│                   │   final_days_lag_days_1.csv
│                   │   final_days_lag_days_3.csv
│                   │   final_days_lag_days_7.csv
│                   │   final_days_lag_hours_1.csv
│                   │   final_days_lag_hours_3.csv
│                   │   final_days_lag_hours_7.csv
│   
└───models
│   │
│   └───results
│   │
│   └───bilstm_multiclass
│   │   │   bilstm_multiclass.ipynb
│   │   │   bilstm_multiclass-tester.ipynb
│   │   │   model.png
│   │
│   └───bilstm_trend
│   │   │   bilstm_trend.ipynb
│   │   │   bilstm_trend-tester.ipynb
│   │   │   model.png
│   │
│   └───cnn_multiclass
│   │   │   cnn_multiclass.ipynb
│   │   │   cnn_multiclass-tester.ipynb
│   │   │   model.png
│   │
│   └───cnn_multiclass
│   │   │   cnn_multiclass.ipynb
│   │   │   cnn_multiclass-tester.ipynb
│   │   │   model.png
│   │
│   └───lstm_multiclass
│   │   │   lstm_multiclass.ipynb
│   │   │   lstm_multiclass-tester.ipynb
│   │   │   model.png
│   │
│   └───lstm_trend
│   │   │   lstm_trend.ipynb
│   │   │   lstm_trend-tester.ipynb
│   │   │   model.png
│   │
│   └───voting_classifier
│       │   voting_classifier.ipynb
│       │   voting_classifiertester.ipynb
│   
└───papers
    │ ALL PAPERS USED

data_generators_cleaners

This folder contains all the scripts required to clean and preprocess the data

BTCprices_cleaner.ipynb - Used to clean the prices dataset. More specifically, it sets the timestamp to UTC and removes the Open, High and Low values.
crypto_prices_getter_yahoo.ipynb - Used to obtain crypto prices from yahoo at different intervals
data_grouper.ipynb - Used to group the lagged dataset hourly or daily
data_grouper.ipynb - Used to group the lagged dataset hourly or daily
data_lag_creator.ipynb - Used to create lagged datasets
english_tweets_extractor.ipynb - Used to remove duplicates and non-English tweets
polarity_adder.ipynb - Used to add polarity and sentiment to tweets
tweets_cleaner_finaliser.ipynb - Used in the last step to remove tweets with less than 4 words after tweets being cleaned.
tweet_processor.py - Used to clean and preprocess tweets

datasets

This folder contains all the datasets All this folder can be found here because it could not be all committed to git due to size

general - Contains the general datasets used to create lagged datasets including the original ones
- with_sentiment - Folder with files containing tweets and their sentiment scores
- BTCDATAwithdate.csv - This contains the cleaned BTC prices dataset
- BTCTWEETS_english_no_duplicates.csv - This contains the dataset of tweets without duplicates and non-English tweets
- preprocessed_tweets.csv - This contains the tweets cleaned and preprocessed from the tweet_processor.py script
- tweets_cleaned.csv - The final cleaned dataset
- BTCDATA.csv - Original dataset for BTC prices
- BTCTWEETS.csv - Original dataset for BTC tweets
tweets_prices - Contains the datasets with each tweet together with the corresponding BTC price
- vader - Datasets with VADER poalrity scores
  - final_days_lag_days_1.csv - Dataset containing uncleaned tweets and prices with 1 day lag
  - final_days_lag_days_3.csv - Dataset containing uncleaned tweets and prices with 3 days lag
  - final_days_lag_days_7.csv - Dataset containing uncleaned tweets and prices with 7 days lag
  - final_days_lag_hours_1.csv - Dataset containing uncleaned tweets and prices with 1 hour lag
  - final_days_lag_hours_3.csv - Dataset containing uncleaned tweets and prices with 3 hours lag
  - final_days_lag_hours_7.csv - Dataset containing uncleaned tweets and prices with 7 hours lag
  - cleaned - Contains cleaned datasets
    - final_days_lag_days_1.csv - Dataset containing cleaned tweets and prices with 1 day lag
    - final_days_lag_days_3.csv - Dataset containing cleaned tweets and prices with 3 days lag
    - final_days_lag_days_7.csv - Dataset containing cleaned tweets and prices with 7 days lag
    - final_days_lag_hours_1.csv - Dataset containing cleaned tweets and prices with 1 hour lag
    - final_days_lag_hours_3.csv - Dataset containing cleaned tweets and prices with 3 hours lag
    - final_days_lag_hours_7.csv - Dataset containing cleaned tweets and prices with 7 hours lag
tweets_prices_volumes_sentiment - Contains the grouped datasets with averaged polarity scores for that time interval with the corresponding BTC prices
- vader - Datasets with VADER poalrity scores
  - day_datasets - Datasets grouped daily
    - final_days_lag_days_1.csv - grouped uncleaned dataset with 1 day lag
    - final_days_lag_days_3.csv - grouped uncleaned dataset with 3 days lag
    - final_days_lag_days_7.csv - grouped uncleaned dataset with 7 days lag
    - final_days_lag_hours_1.csv - grouped uncleaned dataset with 1 hour lag
    - final_days_lag_hours_3.csv - grouped uncleaned dataset with 3 hours lag
    - final_days_lag_hours_7.csv - grouped uncleaned dataset with 7 hours lag
    - cleaned - Contains cleaned datasets
      - final_days_lag_days_1.csv - grouped cleaned dataset with 1 day lag
      - final_days_lag_days_3.csv - grouped cleaned dataset with 3 days lag
      - final_days_lag_days_7.csv - grouped cleaned dataset with 7 days lag
      - final_days_lag_hours_1.csv - grouped cleaned dataset with 1 hour lag
      - final_days_lag_hours_3.csv - grouped cleaned dataset with 3 hours lag
      - final_days_lag_hours_7.csv - grouped cleaned dataset with 7 hours lag

models

This folder contains all the models

bilstm_multiclass - Contains the implementation of the bilstm model which predicts the magnitude of the next day's closing price change
- bilstm_multiclass.ipynb - The actual model implementation
- bilstm_multiclass-tester.ipynb - Script which tests various combinations of hyperparameters to obtain comparable results
- model.png - Model's figure
bilstm_trend - Contains the implementation of the bilstm model which predicts the direction of the next day's closing price
- bilstm_trend.ipynb - The actual model implementation
- bilstm_trend-tester.ipynb - Script which tests various combinations of hyperparameters to obtain comparable results
- model.png - Model's figure
cnn_multiclass - Contains the implementation of the cnn model which predicts the magnitude of the next day's closing price change
- cnn_multiclass.ipynb - The actual model implementation
- cnn_multiclass-tester.ipynb - Script which tests various combinations of hyperparameters to obtain comparable results
- model.png - Model's figure
cnn_trend - Contains the implementation of the cnn model which predicts the direction of the next day's closing price
- cnn_trend.ipynb - The actual model implementation
- cnn_trend-tester.ipynb - Script which tests various combinations of hyperparameters to obtain comparable results
- model.png - Model's figure
lstm_multiclass - Contains the implementation of the lstm model which predicts the magnitude of the next day's closing price change
- lstm_multiclass.ipynb - The actual model implementation
- lstm_multiclass-tester.ipynb - Script which tests various combinations of hyperparameters to obtain comparable results
- model.png - Model's figure
lstm_trend - Contains the implementation of the lstm model which predicts the direction of the next day's closing price
- lstm_trend.ipynb - The actual model implementation
- lstm_trend-tester.ipynb - Script which tests various combinations of hyperparameters to obtain comparable results
- model.png - Model's figure
voting_classifier - Contains the implementation of the voting classifier
- voting_classifier.ipynb - The actual model implementation
- voting_classifier-tester.ipynb - Script which tests runs the classifier several times

papers