Stop crawling websites at some point
Closed this issue · 1 comments
XeniaRieger commented
If wikipedia etc. was cralwed x times stop
LarsMaronde99 commented
Look into self.__crawl_state dict. Add a new key and count how often the site has been crawled