/Data-Engineering-Project-RicknMorty

Rick n Morty - Data Engineering project: pipeline development Bulk and incremental loads, CDC

Primary LanguagePython

The Rick and Morty API

Data-Engineering-Project-RicknMorty

Rick&Morty_API (python request) --> Flatefile (CSV) --> Database (sqlLite or Postgres)

Source API Data from: https://rickandmortyapi.com/

Project Plans (may be modified):

  • v1 - Rick n Morty - Data Engineering project: ETL pipeline development Bulk file loading to postgres/sqlite db.
  • v2 - following with incremental loads, or CDC.
  • v3 - opensouce analytics library for data viz (initally Jupyter)
  • v4 - Local Application - Cron job (linux (shell script) & window (bash script) version and Terminal menu options w/ commandline arguments
  • v5 - Software Architecture pattern update (OOP & code packaging - folders, function daisy chaining) and Dependecy injection for decoupling on small scale.
  • v6 - Make a frontend UI consumer to interact with backend (SQL & Flaskapp) and embedded analytics (GeoJson/ Shapefiles)
  • v7 - deploy as cloud app (example) w/ Heroku, digital ocean, or vercel.