News scraper go

This repository contains the code of the workers that extract data from newspapers el mundo, el pais, la vanguardia and la razón.

Installation

You will also need golang (version 1.12 at least). Follow this to install it.

Configuration

Clone this repository to a directory:

git clone https://github.com/news-scrapers/news-scraper-workers-go.git
Move to the cloned directory and create a file named .env . Inside this file you will need to specify the url to the mongodb database and the newspapers that you want to scrap. Also an id for your scraper. Here is an example that will work if you are running the backend locally and if you want to scrap all newspapers:
```
  scraper_id=scraper_test
  device_id=device_test
  app_id=all
  newspaper=all
  scraping_date_limit=2020-03-10
  scraper_type=daily
  database_url =mongodb://localhost:27017
  database_name=news-scraped-with-tags
```

if you only want to scrap elmundo y el pais you should replace newspaper=all with newspaper=elpais;elmundo (add them separated by ";"). Right now you can scap the following:

elpais
elmundo
abc
diario.es
publico
lavanguardia
20minutos

if you use scraper_type=daily in .env you will scrap the last news in each newspaper, if you want to scrap all historic news you should use scraper_type=historic, the scraper will stop when it reaches the limit date for scraping specified in scraping_date_limit=2020-03-10. Historic scraping is only available for:

elpais
elmundo
abc
lavanguardia

Run the golang code. Inside the project folder run:
```
  go run main.go
```
you should see the logs with the scraped news of the different newspapers.

Alternatively you can use docker-compose:
```
  docker-compose up
```
Or
```
  docker-compose up -d 
```

news-scrapers/news-scraper-workers-go

News scraper go

Installation

Configuration