/public-data-crawler

Crawl vast oceans of html and assemble csvs

Primary LanguageJavaScript

public-data-crawler

Crawl vast oceans of html and assemble csvs.

###Installation Install nodejs, phantomjs and casperjs and then install local node and bower dependencies

###To crawl Placer County Assessor

casperjs --ignore-ssl-errors=true placer-crawler.js

* Requires data is available in a JSON array called placer-data.json. That data will be updated with information from the site. See [Sample input data](#Sample input data) *

###To crawl San Luis Obispo County Assessor

casperjs san-luis-obispo-crawler.js

* Requires data is available in a JSON array called san-luis-obispo-crawler.json. That data will be updated with information from the site. See [Sample input data](#Sample input data) *

###To create CSVs of the crawled data

node create-csv san-luis-obispo-data.json san-luis-obispo-data.csv

###Sample input data

[{"id": "123456789"},{"id": "123456790"}]