A scalable admin ui for scrapy spider service
ScrapyKeeeper is a fork of SpiderKeeper
Forked to provide:
- Enhanced statistics (Errors, Exceptions, Retries...)
- Cache support
- Dashboarding
- Most of pending SpideKeeper ahead commits
How to install ?
git clone https://github.com/fliot/ScrapyKeeper.git
cd ScrapyKeeper
pip install .Running:
scrapykeeper -h
Usage: scrapykeeper [options]
Admin ui for scrapy spider service
Options:
-h, --help show this help message and exit
--host=HOST host, default:0.0.0.0
--port=PORT port, default:5050
--username=USERNAME basic auth username ,default: admin
--password=PASSWORD basic auth password ,default: admin
--type=SERVER_TYPE access spider server type, default: scrapyd
--server=SERVERS servers, default: ['http://localhost:6800']
--database-url=DATABASE_URL
ScrapyKeeper metadata database default: sqlite://./ScrapyKeeper.db
--no-auth disable basic auth
--no-sentry disable sentry.io error reporting
-v, --verbose log level