/crawlab-lite

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Primary LanguageVueBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

Crawlab Lite

中文 | English

Lite version of Crawlab, golang-based web crawler management platform, supporting crawlers in any language.

Compared with Crawlab, this lite version focuses on crawler management on a single machine, it runs independent of any external database and removes a lot of non-essential features.

⚠️ This version is still in early development and some features may be unstable.

Quick Start

Docker Compose

  1. Create docker-compose.yml in any directory as follows:
version: '3'
services:
  master:
    image: zkqiang/crawlab-lite:latest
    container_name: master
    ports:
      - "8080:8080"
    volumes:
      - "./data:/app/data"  # persistent volume
  1. Run the command in this directory:
docker-compose up -d
  1. Visit http://localhost:8080

Source Code

  1. Clone repository
git clone https://github.com/crawlab-team/crawlab-lite
cd crawlab-lite
  1. Run backend
cd backend
go run main.go
  1. Run frontend
cd ../frontend
npm i && npm run serve
  1. Visit http://localhost:8080

Screenshot

Spider List

Task List

Schedule List

Comparison with Crawlab

Crawlab Lite Crawlab
Cross-language spider ✅ ✅
Multi node ❌ ✅
Cron ✅ ✅
Log output ✅ ✅
Spider version manage ✅ ❌
Analytics ❌ ✅
Notification ❌ ✅
Online code editor ❌ ✅
Configurable spider ❌ ✅
SDK ❌ ✅