This project processes real-time retail transactions data from a Kafka topic "transactions" using Spark Streaming. Finally, the processed streaming data is loaded into Snowflake for further analysis and reporting.
The project is designed with the following system architecture:
The Kafka Producer component is responsible for publishing transaction data to the "transactions" topic. It acts as the data source for the Spark Streaming Consumer.
The Spark Streaming Consumer component subscribes to the "transactions" topic in Kafka. It consumes the incoming data stream in real-time, performing the following operations:
-
Decoding: The consumer decodes the incoming data, ensuring it is in a usable format for further processing.
-
Data Cleaning and Type Conversion: The Spark Streaming application performs data cleaning operations,converting data types as needed.
-
Data Loading: The processed data is loaded into Snowflake, which serves as the data warehouse for further analysis and reporting.
-
Schema Definition: The schema enables Spark to treat the streaming data as a structured DataFrame
Snowflake is utilized as the data warehouse for storing and managing the processed transaction data.
-
Englis Video
-
Arabic Podcast
Feel free to connect with me on LinkedIn