This project is the backend side of Brave News, and it fetches the articles from the Brave-defined publishers and shows their feeds/News in the Browser.
For more details: https://brave.com/brave-news-updates/
Required setup:
virtualenv -p /usr/bin/python3.9 .venv
. .venv/bin/activate
pip install -r requirements.dev.txt
To generate sources and list of feeds:
export PYTHONPATH=$PWD:$PWS/src
NO_UPLOAD=1 NO_DOWNLOAD=1 python src/csv_to_json.py
To generate browser feed and images:
export PYTHONPATH=$PWD:$PWS/src
NO_UPLOAD=1 python src/feed_processor_multi.py feed
To update the favicon urls:
export PYTHONPATH=$PWD:$PWS/src
NO_UPLOAD=1 NO_DOWNLOAD=1 python src/favicons_covers/update_favicon_urls.py
We configured the pre-commit hooks to ensure the quality of the code. To setup the pre-commit hooks run the following commands:
pre-commit install
pre-commit run --all-files
This service organizes as follows:
news_aggregator/
├── bin/ # This dir contains the helping shell scripts for Dockerfile.
├── lib/ # This dir contains the utility modules.
├── models/ # This dir contains the dataclasses.
├── sources/ # This dir contains the sources files.
├── src/ # This dir contains all the python script to run the new aggregator.
├── tests/ # This dir contains the tests.
The wasm_thumbnail.wasm
binary comes from https://github.com/brave-intl/wasm-thumbnail.