/football-prediction-crawler

Primary LanguageHTML

football-prediction-crawler

Installation

  1. Install Scrapy

    $ pip install scrapy

  2. Install Docker

  3. Pull the image

    $ sudo docker pull scrapinghub/splash

  4. Install scapy-splash

    $ pip install scrapy-splash

  5. Start the container

    $ sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash

  6. Splash is now available at 0.0.0.0 at ports 8050 (http) and 5023 (telnet).

Crawl data

Firstly, walk into PredictionSpyder and enable Splash

$ cd PredictionSpyder
$ sudo docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash

If you want to crawl winner prediction from google, you need to enable splash to render javascript

$ scrapy crawl winnerspider -o winner.json -t json

==================================================

Or keonhacai prediction

$ cd PredictionSpyder
$ scrapy crawl keonhacaispider -o keonhacai.json -t json

==================================================

Or 188 bet prediction, to update ALL MATCHES, run

$ cd PredictionSpyder
$ scrapy crawl 188spider -o 188.json -t json

At the time World Cup happens, we could pass a url to crawler

$ cd PredictionSpyder
$ scrapy crawl 188spider -a default_url='<https://something.com>' -o 188.json -t json

Index database

To update google prediction, run

python PredictionSpyder/neo4j_import.py --source google --method update

To update betodds prediction, run

python PredictionSpyder/neo4j_import.py --source 188bet --method update

To update match score, run

python PredictionSpyder/neo4j_import.py --source matchscore --method update