XeniaRieger/Modern-Search-Engines

Follow robots.txt rules

Closed this issue · 0 comments

XeniaRieger commented

Before adding links to frontier, check if we allowed to crawl.
Follow rules for delay.