Welcome to Django News! This project demonstrates how to use Django to build a news website efficiently. It automates the process of updating news content and allows you to manage everything from your local machine by simply deploying the Django site and updating the database.
To get started, install the required dependencies using the following command:
pip install -r requeriments.txtBefore running the application, set up the configuration file:
export KLEIN_CONFIG=./path/your-config.yml- NewsAPI: For populating the database with news data.
- Requests & BeautifulSoup: For web scraping.
- Selenium: For summarizing news content. (This free option avoids using memory-intensive transformers or paid services like ChatGPT.)
The database should be hosted in the cloud for optimal performance and accessibility. By having your database in the cloud, you ensure that updates and changes are seamless and can be managed directly from your local machine. This setup eliminates the need for long deployment times and allows for quicker updates and maintenance.
- Direct Updates: You can update the database directly from your local machine without needing to redeploy the entire application.
- Efficiency: Reduces deployment times and ensures that changes are reflected immediately.
- Accessibility: Provides easy access to the database from anywhere, facilitating remote management and updates.
To test the database locally, use Docker Compose to create a MongoDB container:
docker-compose up --buildNote: Use config_example.yml as a template to create your own config.yml. The information in config.yml might not be included in the repository.
- 1-) Creating project:
django-admin startproject django_news
cd django_news- 2-) Creating application of news:
django-admin startapp noticiasPopulating the Database with News
python pipeline_populate_db.pyTo scrape news data from the web, execute:
python pipeline_web_scraper.pyTo summarize news content, use:
python pipeline_summarize_text.pyThis part can be an web scraper generating information with OpenAI API.
python pipeline_populate_db.pypython pipeline_web_scraper.pypython pipeline_summarize_text.py- Miguel Angelo do Amaral