/tripadvisor-scraper

Tripadvisor reviews scraper

Primary LanguagePython

TripAdvisor Reviews Scraper

Scraper of Tripadvisor reviews, parametric by date and language. The script allows to scrape:

  1. urls of TA points of interests based on string query
  2. POIs metadata
  3. POIs reviews up to a certain minimum date and with a specified language

Installation

Follow these steps to use the scraper:

  • Download Chromedrive from here.

  • Install Python packages from requirements file, either using pip, conda or virtualenv:

      `conda create --name scraping python=3.6 --file requirements.txt`

Note: Python >= 3.6 is required.

Usage

The scraper has 5 parameters:

  • --i: input file, containing a list of Tripadvisor urls that point to first page of reviews.
  • --lang: language code to filter reviews. Note: only "select all languages" click is implemented.
  • --N: number of reviews to scrape.
  • --q: string query to scrape url places.
  • --place: boolean value to scrape place metadata instead of reviews.

Some examples:

  • python scraper.py --q amsterdam: generates the urls.txt file with the top-30 POIs of amsterdam
  • python scraper.py --place 1: generates a csv file containing metadata of places present in urls.txt
  • python scraper.py: generates a csv file containing reviews of places present in urls.txt

The config.json file allows to set the directory to store output csv, as well as their filenames.

License

GNU GPLv3