Crawler not performing as mentioned in README
drumm91UWO opened this issue · 1 comments
drumm91UWO commented
The Crawler is finding fewer URLs than mentioned in the README for me. I am running Crawl.py but haven't successfully built the 500Lines project. Does that matter? I'm using Windows 10, PyCharm by JetBrains, Python 3.6.1 (because asyncio is required for the crawler to run and asyncio wasn't implemented until Python 3.4).
Specific examples:
Site, distinct URLs mentioned in README, distinct URLs I received
dropbox.com, ~2500, 78
xkcd.com, +1500, 38
drumm91UWO commented
I'm closing this because I spoke to someone who is able to run the crawler. They said once they got the tests to pass, everything was fine.