This is a repository for some simple web-crawler cases during my learning period.
- write script in node.js
- use superagent for web request
- use nedb for persistent storage
- www.tctasia.cn: a basic demo using native http module to get content from a chinese-encoded website.
- www.rci.com: a demo using superagent to crawl deep content.
- www.zhihu.com: a demo simulting user login.
- www.itnint.com: a demo mirroring the website.