This project utilizes Scrapy and Selenium to scrape new movie releases from the Metacritic website. Metacritic is a popular platform for aggregating reviews and ratings for movies, games, music, and TV shows.
The objective of this project is to collect data on new movie releases, including movie title, release date, description, Metascore, and image link, from the Metacritic website.
- Python
- Scrapy
- Selenium
- Clone this repository to your local machine:
git clone <repository_url>
- Install the required Python packages:
pip install scrapy selenium
- Navigate to the project directory:
cd metacritic-movie-scraper
- Run the Scrapy spider to scrape data from the Metacritic website:
scrapy crawl new
-
Setup: The project is set up with Scrapy and Selenium to handle web scraping of dynamic content rendered by JavaScript.
-
Scraping: The Scrapy spider navigates through the Metacritic website, simulating user behavior with Selenium to access dynamically rendered content.
-
Data Collection: For each movie listed on the website, the spider extracts relevant information such as movie title, release date, description, Metascore, and image link.
-
Data Processing: The scraped data is processed and formatted within the Scrapy spider.
-
Saving Data: The formatted data is saved to a CSV file for further analysis and use.
The output CSV file contains the following columns:
- Movie Title
- Release Date
- Description
- Metascore
- Image Link
Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request.
