/webCrawler

Dynamic web crawler,just for single machine

Primary LanguageJava

crawler

##Intro Crawl the infomations of apps from different channel.MyCrawler exposes the restful interface to the users. Users send the trigger infomations(e.g.taksId,channelId and so on),then the crawler can perform perfectly the task and response the apps infos to the user. ##Fetures

  • based on the webmagic0.5.2
  • uniform rest api
  • concurrent crawling approach
  • simpified models
  • high-efficiency HTML parser JSOUP