This project demonstrates a comprehensive data pipeline for ingesting, transforming, and analyzing Olympic data using Azure services. The pipeline involves data ingestion using Azure Data Factory, data storage in Azure Data Lake, transformation using Azure Databricks, analysis using Azure Synapse Analytics, and visualization with Tableau.
The above diagram illustrates the architecture of the data solution:
- Data Source: Olympic data is sourced from a designated source.
- Data Ingestion: Data is ingested into the pipeline using Azure Data Factory.
- Data Lake (Raw Data): The raw data is stored in Azure Data Lake.
- Data Transformation: Data is transformed and cleaned in Azure Databricks.
- Data Lake (Transformed Data): The transformed data is stored in a separate container in Azure Data Lake.
- Analysis: SQL analysis is performed using Azure Synapse Analytics.
- Visualization: Insights are visualized using Tableau.
Azure Data Factory (ADF) is used to orchestrate the data ingestion process. The data is extracted from the source and loaded into the Azure Data Lake.
Azure Data Lake is used for storing both raw and transformed data. It provides scalable and secure storage for big data analytics.
Azure Databricks is used for data transformation and cleaning. It enables collaborative analytics using Apache Spark and other big data technologies.
Azure Synapse Analytics is used for advanced data analytics. It allows for scalable and efficient querying and analysis of large datasets using SQL.
Tableau is used for data visualization. It helps in creating interactive and insightful dashboards to visualize the analysis results.