Repository of python web crawler for course of OSDL.
- Download files into one directory
- run main.py
- if you want to change the address of the site which you want to crawl open main.py and replace whatever the content is there in HOMEPAGE to your desired webpage.