STOCK MARKET DATA ANALYSIS

Use Python to scrape data and join with financial data from Yahoo Finance (or another finance) API. Use data manipulation and visualization for financial and investment analysis (i.e. compare rates of return, calculate risk, build trading algorithms, and make investment decisions).

How to Run

Use the Stock_Market_Data_Analysis.ipynb file to run the program in Jupyter Notebook. Use the ".py" file (Stock_Market_DataAnalysis_DataVisualization.py) to run the program only in Python.
For a walk through of the project - how to install Python, the necessary packages, Jupyter, and run the notebook file - please see my Demo Screen Recording here.
- For users running the project with Jupyter:

After downloading the files, if Python is not installed, please install it by downloading Python here.
After installing Python, open a Python shell and run the following commands to install the necessary packages to run my program:

pip install pandas
pip install pandas-datareader
pip install beautifulsoup4
pip install scikit-learn
pip install numpy
pip install matplotlib
pip install mplfinance
pip install mpl-finance
pip install yfinance
pip install jupyter

Once everything is installed, change directory (cd) to navigate to where the project has been downloaded.
Locate the ".ipynb" file Stock_Market_Data_Analysis.ipynb and run Jupyter with the command: jupyter notebook in your terminal or cmd; this will take you to the project in Jupyter Notebook, opening up a browser.
Once Jupyter opens in the browser, you should see the Stock_Market_Data_Analysis.ipynb notebook file. Double click to open the file.
To run the program, select "Cell," and "Run All."
For additional charts, please run the Stock_Market_Data_Analysis_DataVisualization.ipynb file in Jupyter Notebook. Here I focus mainly on data visualizations for large cap tech stocks (i.e. Apple, Google, Facebook), and model various chart types. I also begin the next part of the project: Fundamental Analysis. In this part, we will be using finanical statements from the SEC Edgar website.

Project Summary

First, this project will focus on technical analysis, measuring stock price data for movement (volatility) and volume.
Second, I plan to expand this project in the future, to include a branch with fundamental analysis, to look more in depth at financial statement analysis.
Third, I am interested in expanding the analysis to include Python for cryptocurrencies, such as financial and investment analysis for ICOs, and predicting crypto prices.

DISCLAIMER: I am not a financial adviser nor a CPA. This project is for educational purposes only. Investing of any kind involves risk. Although it is possible to minimize risk, your investments are solely your responsibility. It is very important to conduct your own research. I am merely sharing my project/opinion with no guarantee of either gains or losses on any investments.

Technical Summary

Back end language: Python (the version used here is Python 3.7.6)
Dataset: csv, stock price data via Yahoo Finance
Packages: Pandas/NumPy; Scikit-learn for Machine Learning in Python; Matplotlib (and mplfinance) for data manipulation and visualization.
(For Code Louisville: 5+ commits using Git.)

Features

The script will scrape data for S&P500 tickers, pull financial data from Yahoo Finance API, and download into a csv file. It will also manipulate/clean data, and merge multiple data frames into one large csv file. The script uses for loops, dictionaries, and error handling. Further, there is additional data visualization in the "Stock_Market_Data_Analysis_DataVisualization.ipynb" Jupyter Notebook file; this is done using matplotlib to build various stock charts (i.e. line charts, bar charts, moving average bar charts, candlestick charts). Additional features are highlighted below:

Scrape stock tickers from web (i.e. Wikipedia).
For this project, the SP500 list is from: List of S&P 500 Companies.
Use Requests to scrape data for Beautiful Soup to parse.
Using this soup object, navigate/search HTML for data you want to pull.
Create directory with stock price close data for all 500 companies over time (i.e. 01/01/2000 to 05/24/2020).
Build quantitative models to conduct financial and investment analysis such as risk and rates of return, and build a basic trading strategy.

Milestones

Import needed packages/modules Main Requirements for this Program:
- pandas
- pandas-datareader
- beautifulsoup4
- scikit-learn
- numpy
- yfinance (or another finance API)
- jupyter
- matplotlib
- mplfinance, mpl_finance

mplfinance requires matplotlib and pandas
Installation: pip install --upgrade mplfinance

You will see the first cell in the Jupyter notebook file will import all the necessary packages to run the program:

import requests
import datetime as dt
import os
import io
import pandas as pd
import pandas_datareader.data as web
import pandas.plotting
from pandas.plotting import register_matplotlib_converters
import pandas.testing #pandas.testing.assert_frame_equal
from pandas.testing import assert_frame_equal #assert_frame_equal

import numpy as np

import matplotlib
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
import matplotlib.dates as mdates
import matplotlib.colors as mcolors
from matplotlib import style

import mplfinance as mpf
import mpl_finance as mplf
from mpl_finance import candlestick_ohlc

import collections
from collections import Counter

import sklearn
from sklearn import svm, neighbors
from sklearn.svm import LinearSVC 
from sklearn.model_selection import train_test_split
from sklearn.ensemble import VotingClassifier, RandomForestClassifier

style.use('ggplot')

Scrape data using BeautifulSoup
Download financial data from API
Join tickers with stock price data
Visualize data using Matplotlib
Create basic trading strategy

Requirements

Build webscraper to get data for S&P500 List.
Use Yahoo Finance (or other) API yfinance for financial data.
Use Pandas to join stock tickers with financial data.
Analyze data with various quantitative models to calculate risk and make investment decisions.
Download data as csv and read.
- Build quantitative models to predict returns & evaluate risk.
- Run basic Value at Risk (VaR) calculations, Monte Carlo Simulations.
- Looking at news sentiment as a proxy for market volatility.
- Graph/visualize data.

NOTE: If you are new to Python, check out the Python Programming Fundamentals website for tutorials. You will need to review up to installing Python packages and modules with pip.

Instructions (Overview):

Step 1: Intro to Using Python for Finance
Step 2: Handling and Graphing Data
Step 3: Stock Data Manipulation
Step 4: Data Resampling
Step 5: S&P500 List Automation
Step 6: Getting S&P500 Stock Price Data
Step 7: Combine DataFrames for S&P500 List and Stock Price Data

Python for Machine Learning

Step 8: Building S&P500 Correlation Table
Step 9: ML: Preprocess Stock Market Data
Step 10: ML: Create Target Function
Step 11: ML: Create Labels
Step 12: ML Algorithm: Mapping Relationships for Stock Prices

The New API (mplfinance)

There is a new matplotlib finance (mplfinance) API that has made creating financial plots easier. Some of the updates include automatic features for the user, and improved interacing with Pandas DataFrames.

Note: This repository is using both modules (the old and new matplotlib/mplfinance APIs), because I am working to update the candlestick charts in the script.
For more information on the old API, please see below.

According to the matplotlib/mplfinance repo, the conventional way to import the new API is as follows:

    import mplfinance as mpf

The most common usage is then to call

    mpf.plot(data)

where data is the Pandas DataFrame object, which contains Open, High, Low and Close pricing data, using a Pandas DatetimeIndex.

Further details on how to call the new API can be found below under Basic Usage on the matplotlib/mplfinance repo, as well as in the jupyter notebooks in the examples folder.