/covid-news

A data engineering personal project for applying some of my skills

Primary LanguagePython

Table of Contents
  1. About The Project
  2. Getting Started
  3. Usage
  4. Contributing
  5. Contact
  6. Acknowledgements

About The Project

This project consistis in a web scraper that collects daily news about the COVID19 pandemic from the BBC Brazil website (https://www.bbc.com/portuguese), store them in a Postgres database and make useful information available through an API.

Built With

  • Python
  • Postgresql
  • Docker

Project Architecture

plot

Getting Started

Prerequisites

  • Docker
sudo apt get docker
  • Docker-compose
sudo apt get docker-compose

Installation

  1. Clone the repo
git clone https://github.com/brianamaral/covid-news.git
  1. Change de environment variables in the docker-compose.yml
- POSTGRES_USER=
- POSTGRES_PASSWORD=
- POSTGRES_DB=
- POSTGRES_ADDRES=

Usage

If you changed the variables above correctly, then you are good to go, just go to the docker folder and then run:

docker-compose up

API Endpoints

  • /all_articles: return all articles on the dabatase
  • /last_article: return the last articles on the database by the date

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

Contact

Brian Bomfim Amaral - Linkedin - brian.amaralt@gmail.com

Project Link: https://github.com/brianamaral/covid-news

Acknowledgements