Created this pipeline which orchestrates a daily ETL pipeline for news data using Python. Utilizing the News API, it fetches top headlines, cleans, and transforms the data. The processed information is then uploaded to AWS S3 using Boto3. The DAG runs daily, with error retries and email notifications configured.
- Check Python version
- Make sure
Python 3.11.5
is installed - To check your python version.
python -V
-
Install dependencies
- Use the
requirements.txt
file to install all required dependencies.
pip install -r requirements.txt
- Use the
-
Start Docker Desktop
-
Pull my docker Images from the
ronakpanchalk9/ronak-news-api
repository on DockerHub. Using:docker pull ronakpanchalk9/ronak-news-api
-
Run the
docker-compose.yaml
file,
docker-compose up -d
-
-
Config file
- Create a
config.ini
file and store it in the same directory and insert the variables in it; config.ini [api] api_key = [Enter the API key here] [storage] region_name = [Enter the region name] aws_access_key_id = [Enter your access key] aws_secret_access_key = [Enter your Secret access key] bucket_name = [Enter your bucket name]
- Create a
The DAG is designed to run daily, triggered by Airflow's scheduler. The process begins by fetching news data from the News API, followed by cleaning and transforming the data using pandas. The final step involves uploading the processed data to an AWS S3 bucket, ensuring a secure and scalable storage solution.
Automation: The daily scheduling ensures a consistent and automated flow of fresh news data, eliminating the need for manual intervention.
Scalability: Leveraging AWS S3 provides scalable storage, accommodating growing datasets with ease.
Flexibility: Airflow's DAG structure allows easy modification and expansion of the pipeline, adapting to evolving requirements.
Error Handling: Robust error-handling mechanisms are integrated into each step, providing insights into any issues that may arise during data retrieval, transformation, or upload.