RuedigerVoigt/exoskeleton
A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend
PythonApache-2.0
Issues
- 1
Replace the requests with aiohttp or httpx
#26 opened by RuedigerVoigt - 0
- 0
- 0
Improve estimate of remaining time
#24 opened by RuedigerVoigt - 1
Python 3.9 support
#22 opened by RuedigerVoigt - 0
Create a Docker image
#21 opened by RuedigerVoigt - 0
- 1
Add Integration Test with MariaDB
#15 opened by RuedigerVoigt - 1
Add ability to block domains
#16 opened by RuedigerVoigt - 0
Duplicate tasks in the queue
#17 opened by RuedigerVoigt - 0
Add support for a remote Mailserver
#14 opened by RuedigerVoigt - 1
Cloud services as a storage option
#9 opened by RuedigerVoigt - 1
Add to CI pipeline
#2 opened by RuedigerVoigt - 0
- 1
- 0
Add base URL to links
#13 opened by RuedigerVoigt - 1
Python 3.8 Test
#4 opened by RuedigerVoigt - 1
colored log output
#11 opened by RuedigerVoigt - 1
PostgreSQL support
#3 opened by RuedigerVoigt - 0
add statistics on host basis
#8 opened by RuedigerVoigt - 3
- 0
Unit Tests
#1 opened by RuedigerVoigt - 0
File Name Prefix
#10 opened by RuedigerVoigt - 0
add crawl delay in case of timeout
#7 opened by RuedigerVoigt