/PIB-Data-Analysis

End to end Batch pipeline and Data Analysis about Brazilian economics.

Primary LanguageJupyter NotebookMIT LicenseMIT

PIB-Data-Analysis

End-to-End data pipeline and Analysis about Brazilian economics.

Progress checklist:

  • Create GCS buckets and config
  • Design architecture diagram (private until finish the project)
  • Set development environment
  • Create pre-processing python script
  • Create processing pyspark script
  • Implement airflow DAGs
  • Implement Bigquery Data Warehouse
  • Create PowerBI Dashboard
Architecture Diagram ( Work in Progress )
space-1.jpg
I made this diagram using Diagrams.net.