/web-crawler

Primary LanguagePython

task1: 
use BFS to crawl the seed 'https://en.wikipedia.org/wiki/Sustainable_energy' 
max depth = 5
max num of url = 1000

task2-BFS: 
use BFS to crawl the seed 'https://en.wikipedia.org/wiki/Sustainable_energy' 
keyword ’solar’
max depth = 5
max num of url = 1000

task2-DFS: 
use DFS to crawl the seed 'https://en.wikipedia.org/wiki/Sustainable_energy' 
keyword ’solar’
max depth = 5
max num of url = 1000

task3: 
merge two crawl results
https://en.wikipedia.org/wiki/Sustainable_energy' 
'https://en.wikipedia.org/wiki/Solar_power'
rank url