richardpenman/asyncrawler

Asyncio based web crawling framework

Features

• Asynchronous downloading using aiohttp
• Downloads cached locally in sqlite
• Continue an interrupted crawl
• Proxies
• Cookies
• Handle redirects
• Retry 5XX errors

Example

>>> import asyncrawler
...

Install

Install from pypi:

pip install asyncrawler

Or checkout latest version from repository:

git clone https://github.com/richardpenman/asyncrawler