hueristiq/xcrawl3r

A command-line interface (CLI) based utility to recursively crawl webpages. It is designed to systematically browse webpages' URLs and follow links to discover linked webpages' URLs.

GoMIT

Issues

Support cookies during the crawl [ENHANCEMENT]
#16 opened 7 months ago by Zabrane
1
【小想法】加入blacklist选项进行爬取格式筛选，例如过滤掉jpg，css链接
#15 opened 7 months ago by givemefivw
0
How to batch crawl？
#12 opened a year ago by zglaozhu
6
Add JS rendering
#4 opened 2 years ago by enenumxela
0
Parse URLs JavaScript instead of rendering
#1 opened 2 years ago by enenumxela
2