magic890/tripadvisor-scraper

problem running it

anupamme opened this issue · 4 comments

anupamme commented

I installed scrapy on mac os 10.9.5 (maverics) as scrapy-2.7. Then tried running this:

scrapy-2.7 crawl tripadvisor-restaurant -o output/result.json -t json

Issues I am facing.

  1. It has no command command crawl

So I tried running

scrapy-2.7 fetch tripadvisor-restaurant -o output/result.json -t json

Then I get this error: no such option: -o

So I am not sure what I am doing wrong.

magic890 commented

Are you into the folder tripadvisor-scraper (the one with inside the folder output)?
Here you can run: scrapy crawl tripadvisor-restaurant -o output/result.json
I'm running OS X 10.10 with Scrapy 0.20.2, and it's works fine.

For crawl other links, you have to edit:
tripadvisorbot/spiders/tripadvisor-restaurant.py:18

Let me know if this works.
In this day I'll test this script using scapy-2.7.

feliu commented

the script doesnt work in scapy-2.7

magic890 commented

Hi @feliu,

I can't find Scrapy 2.7 version, the latest seems to be Scrapy 0.24.4, so I'm guessing that you're running the latest Scrapy version on Python 2.7.

Now I'm testing this script on Mac OS X 10.10.2 running Python 2.7.6 + Scrapy 0.24.4 and it's works fine.
The only issue is that TripAdvisor.com DOM has changed since my last commit, so script will generate an invalid JSON output file.
I'll push a fix for XPath DOM queries as soon as possible in the next few days.