/kongfzCrawler

an old project which is used to crawl bookshop information from a website named 孔夫子, the website is http://www.kongfz.com/

Primary LanguageJavaMIT LicenseMIT

kongfzCrawler

An old project which is used to crawl bookshop information from a website named 孔夫子, the website is http://www.kongfz.com/

This crawler was manually created rather than using general web crawler framework but ip banning problem was perfectly solved by using network reconnection. The only condition is that there must be an ip pool for us to change ip everytime when reconnection happens.