Data pipeline project
This project aims to work on different aspects of a data pipeline. It is meant to learn new technologies, but hopefully it also helps other people to get inspired. Note that this repository will (slowly) change and get more functionality.
Objectives
The fictive story I will base this repository on is Mario, since I like to use a theme to work on my project. I will gather some character data and create events to fake the characters moving through different Mario worlds. To do this I will use Python (code), Kafka (Spark streaming) and minIO (object storage). All other tools will follow during the implementation of this project.
- Data crawler to get character data
- Data connection with minIO storage
- Event generator to create the (fake) events
- Kafka producer to put events on the Kafka bus
- Spark Stream reader to ingest the events
- Spark Stream parser to combine ingested data with the character data
- Dashboard to visualize the events
Setting up virtual environment
I use pipenv
with Python version 3.7.3 to setup the environment.
$ pipenv --python=3.7
Creating a virtualenv for this project…
Pipfile: /Users/jitsejan/code/data-pipeline-project/Pipfile
Using /Library/Frameworks/Python.framework/Versions/3.7/bin/python3 (3.7.3) to create virtualenv…
⠸ Creating virtual environment...Using base prefix '/Library/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/jitsejan/.local/share/virtualenvs/data-pipeline-project-DUniwI99/bin/python3
Also creating executable in /Users/jitsejan/.local/share/virtualenvs/data-pipeline-project-DUniwI99/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
✔ Successfully created virtual environment!
Virtualenv location: /Users/jitsejan/.local/share/virtualenvs/data-pipeline-project-DUniwI99
Creating a Pipfile for this project…
$ pipenv shell
Launching subshell in virtual environment…
. /Users/jitsejan/.local/share/virtualenvs/data-pipeline-project-DUniwI99/bin/activate
Installing libraries
Note that pipenv install
should be used rather than pip
or pip3
install when using pipenv
.
$ pipenv install lxml
Installing lxml…
Adding lxml to Pipfile's [packages]…
✔ Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (b70304)!
Installing dependencies from Pipfile.lock (b70304)…
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:01
Testing code
The following will run the tests and the coverage check simultaneously.
$ pytest --cov=. -v
Convert the results to HTML for better visiblity on the coverage results.
$ coverage html
$ open htmlcov/index.hml