/Stock-Market-Real-Time-Data-Engineering-Project

In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka. We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

Primary LanguageJupyter Notebook

Stock-Market-Real-Time-Data-Engineering-Project

Introduction

In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka.

We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

Architecture

Technology Used

  • Programming Language - Python
  • Amazon Web Service (AWS)
  1. S3 (Simple Storage Service)
  2. Athena
  3. Glue Crawler
  4. Glue Catalog
  5. EC2
  • Apache Kafka

Dataset Used

You can use any dataset, we are mainly interested in operation side of Data Engineering (building data pipeline)

Here is the dataset used - https://github.com/mihirkudale/Stock-Market-Real-Time-Data-Engineering-Project/blob/main/indexProcessed.csv