See project writeup at ProjectWriteUp.pdf Requirements -url-normalize-1.3.1 for python2.6 -lucne version 6.4.1 for python2.7 -HTML Tidy for Mac OS X version 5.4.0 -pytidylib-0.3.2 for python2.6 -robotparser 13.3 for python3
To run -change port number at line 68 in testRetriever http://linserv1.cims.nyu.edu:45457/ python2.7 testRetriever.py