mittenchops/microcrawler

Simple node.js web scrapping engine ...

microcrawler

Status

Prerequisites

• node.js
• npm
• grunt

Getting started

Just clone.

# Clone repository
git clone https://github.com/korczis/microcrawler.git

# Enter folder
cd microcrawler

# Install required packages
npm install

# Run tests
grunt

Example usage

Craiglist

node app.js -p craiglist.listing http://sfbay.craigslist.org/sfc/sss/

Google

node app.js -p google.listing http://google.com/search?q=Buena+Vista

Hacker News

node app.js -p hackernews.index https://news.ycombinator.com/

xkcd

node app.js -p xkcd.listing http://xkcd.com

Yelp

node app.js -p yelp.listing "http://www.yelp.com/search?find_desc=restaurants&find_loc=Los+Angeles%2C+CA&ns=1&ls=f4de31e623458437"

Youjizz

node app.js -p youjizz.listing http://youjizz.com

Credits

• @pavelbinar for QA and not just that.