(Remark: Currently, this is a Work in progress)
I wish to create a stock database that scrape the price and finanncial statement data to the MySQL database.
Below is the ER Diagram of this database project:
- python + mysql database
- rename .sample_env to .env, and then rep
- open terminal, and then type
python -m app.database.db_controller - check back your mysql gui to see if the database is created
- CSV file for FinancialStatementLineSequenceDB: https://github.com/leosmigel/analyzingalpha/tree/master/2020-02-28-financial-statement-database
- use intrinio as a baseline for database design of macro economics data: https://docs.intrinio.com/documentation/web_api/get_all_economic_indices_v2
- use schedule + threading to repeat scraping process at any interval: https://www.geeksforgeeks.org/python-schedule-library/ & https://schedule.readthedocs.io/en/stable/parallel-execution.html
- use numba + numpy to replace pandas for faster iteration
- learn pytest for unit testing and Integration testing:
- (i) Unit testing: https://github.com/OpenBB-finance/OpenBBTerminal/blob/develop/tests/README.md
- (ii) Integration testing: https://github.com/OpenBB-finance/OpenBBTerminal/blob/develop/openbb_terminal/miscellaneous/integration_tests_scripts/README.MD
Build a MySQL database which scrape historical stock price data, news data, and macroeconomics data.
-
Makefile to automate the mysql db creation... (+Github actions)
-
Use financedatabase to get data: https://pypi.org/project/financedatabase/
database migrations tool: alembic
- Step 1: Type
alembic init alembicat your working directory. It will create an folder namedalembic. Skip this ifalembicfolder exists - Step 2: Type
alembic revision --autogenerate - Step 3: Type
alembic upgrade head