Unveiling the intricacies of political discourse, the Politifact Web Scraping Project is a Python-powered endeavor utilizing the Scrapy framework. This initiative seeks to extract and analyze fact-checking data from Politifact.com, providing valuable insights into political statements, rulings, and the evolving information landscape.
- Data Extraction: Scraps author names, saying dates, headlines, rulings, publishers, and article URLs for a comprehensive dataset.
- File Management: Dynamically creates directories for organized storage of scraped data, ensuring a systematic approach from the project's outset.
- Image Downloads: Utilizes Scrapy's image pipeline for downloading header images, enhancing the visual context of each article.
- Efficient CSV Handling: Implements regular write intervals to prevent data loss and alleviate memory burden during asynchronous requests.
- Python 3.x
- Scrapy
- Requests
- Pandas
-
Clone the Repository:
git clone https://github.com/Muneeb1030/WebScrapper_Politifact.git -
Install Dependencies:
pip install scrapy pandas requests -
Run the Scraper:
scrapy crawl politifact
- Customization:
- Tailor the scraper to your needs by modifying the Scrapy spiders.
- GitHub Repository:
- Explore, contribute, and stay updated on the GitHub repository.
This project is intended for educational purposes and strictly adheres to Politifact's terms of service. Users are advised to deploy the scraper responsibly and in compliance with platform policies.
Explore the project in detail through my Medium blog, where I share insights, motivation, and in-depth explanations about the Politifact Scraper.
- M Muneeb ur Rehman
Feel free to fork, contribute, and enhance the capabilities of this Politifact scraper. Happy scraping! 🌐💻