/crode.js-Node-web-Crawler

DISCLAIMER: old stuffs, not maintained anymore. Might see spiders and noodles if ya jump into it

Primary LanguageJavaScriptGNU General Public License v3.0GPL-3.0

crode.js

Visit haxzie.me for demo.

Node.js web crawler scrapping file downloadable link from open ftp servers. crode.js is an express app.crawling is made possible using npm - Crawler. app scrapes the html file using npm modules:

-request

-cheerio

Storage

crode.js uses Mongo db by default currenty uses only two db schemas for storing download links found and visited.