/hk0weather

Web scraper project to collect the useful Hong Kong weather data from HKO website

Primary LanguagePython

hk0weather codecov

hk0weather

hk0weather is an open source web scraper project using Scrapy to collect the useful weather data from Hong Kong Observatory website.

Scrapy can output collected weather data into the machine-readable formats (eg. CSV, JSON, XML).

Available Web Crawlers

  1. regional: Hong Kong Regional Weather Data in 10-minutes update from HKO.
  2. rainfall: Hong Kong Rainfall Data in hourly update from HKO.
  3. hkoforecast: Hong Kong Next 24 hour Weather Forecast Report from HKO Open Data.
  4. hko9dayforecast: Hong Kong 9-day Weather Report from HKO Open Data.

Installation

Cloning and setup hk0weather in a Py3 virtual environment

$ git clone https://github.com/sammyfung/hk0weather.git
$ cd hk0weather
$ python3 -m venv venv
$ source venv/bin/activate  
$ pip install -r requirements.txt

Run a Scrapy spider

Activate the Py3 virtual environment once before the first running of web spiders.

$ source venv/bin/activate  
$ cd hk0weather

Optionally, list all available spiders.

$ scrapy list

Run a regional weather data web crawler and export data to a JSON file.

$ scrapy crawl regional -o regional.json

References