Scrappy
Just a fun project to understand what a web crawler does...
Installation
- Clone this repo
- Install python3.
- Fill in PROJECT_NAME and BASE_URL of your required website and save it.
- Open terminal in the repos folder and type
python main.py
Output
- Open the required project folder that is generated once you run the main.py
- "crawled.txt" contains all the crawled links.