scrapyd
There are 66 repositories under scrapyd topic.
Gerapy/Gerapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
my8100/scrapydweb
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:
DormyMo/SpiderKeeper
admin ui for scrapy/open source scrapinghub
kezhenxu94/house-renting
Possibly the best practice of Scrapy 🕷 and renting a house 🏡
baabaaox/ScrapyDouban
豆瓣电影/豆瓣读书 Scarpy 爬虫
casual-silva/NewsCrawl
狠心开源企业级舆情新闻爬虫项目：支持任意数量爬虫一键运行、爬虫定时任务、爬虫批量删除；爬虫一键部署；爬虫监控可视化; 配置集群爬虫分配策略；👉 现成的docker一键部署文档已为大家踩坑
mouday/spider-admin-pro
spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看 和 爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版
my8100/files
Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects
crawlab-team/crawlab-lite
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
bitmakerla/estela
estela, an elastic web scraping cluster 🕸
my8100/scrapyd-cluster-on-heroku
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:
aaldaber/Distributed-Multi-User-Scrapy-System-with-a-Web-UI
Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner
my8100/logparser
A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.
datawizard1337/ARGUS
ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9
EasyPi/docker-scrapyd
🕷️ Scrapyd is an application for deploying and running Scrapy spiders.
ConlinH/aio-scrapy
Implement scrapy with asyncio
fliot/ScrapyKeeper
admin ui for scrapy/open source scrapinghub
jxltom/scrapymon
Simple Web UI for Scrapy spider management via Scrapyd
liangWenPeng/scrapy-admin
A django admin site for scrapy
ScrapeOps/scrapeops-scrapy-sdk
Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.
ToonoW/SpiderManager
爬虫管理平台
jxltom/scrapyd-heroku
Wrapper for running Scrapyd in Heroku or locally as a service
q-m/scrapyd-k8s
Scrapyd on container infrastructure
Tiago-Lira/scrapyd-mongodb
Library designed to replace the SQLite backend by a MongoDB backend on Scrapy queue management
heidudu/tophub
后端基于Python的Flask和Scrapy,前端基于React,redux，采用docker部署的资讯收集站
perrornet/SpiderMan
SpiderMan Based on Scrapy, scrapyd, scrapy-API, tornado spider distributed management framework.
WangPei0316/scrapy-zhihu-user
知乎用户爬虫，使用scrapy_redis，scrapyd，gerapy等
harrisbaird/dockerfiles
Various Dockerfiles.
my8100/scrapyd-cluster-on-heroku-scrapyd-app
How to set up Scrapyd cluster on Heroku
Dainius-P/scrapyd-dash
Scrapyd Dashboard
INNOVINATI/scrapretty
A pretty and serverless dashboard for your Scrapyd instances
mpszumowski/djangocrawler
start a new (random) life | World Bank data scraper on django rest app
zubedev/scrapydoo
Scrapy dappy doo crawler for proxy sites
Benknightdark/ScrapydDashboard
Scrapyd 爬蟲儀表板
f213/scrapyc
CLI and client library for scrapyd. Done right.
Dog-Egg/scrapy-ui
A web service that manages and schedules Scrapyd. 🕷️