codeGUST-SE/crawler

ignore the URL tree

mustiikhalil opened this issue · 1 comments

mustiikhalil commented

Need to make the crawler ignore the Url tree and only crawl blob/master/

mustiikhalil commented

Fixed it and done