Crocodile Distributed Task Scheduling System

English | 中文

Introduction

A distributed task scheduling system based on Golang that supports http requests, run golang、python、python3、shell、bat

Screenshot

Click Me

                                                  +----------+
        +-------------+                           ||--------||
        ||-----------||                           ||        ||
        ||           ||                           || Worker ||
        ||   调度中心 ||                           ||         ||
        ||           ||      RPC调用(gRPC)        ||---------||
        ||-----------|| +-----------------------> ||-------- ||
        ||-----------||                           ||        ||
        ||           || <-----------------------+ || Worker ||
  +---> ||   调度中心 ||      任务实时日志            ||        ||
任 |    ||           ||                           ||--------||
务 |    ||-----------|| <------+                  ||--------||
完 |     +------------+        |                  ||        ||
成 |         ^    |            |                  || Worker ||
持 |       实|任   |            |数                ||        ||                
久 |       时|务   |获          |据                 +----------+ 
日 |       志|状   |取          |存                
志 |       日|态   |锁          |储
   |        v     v            v
   |      +-------+-+      +----+---+
   |      |         |      |        |
   +----- |  Redis  |      | MySQL  |
          |         |      |        |
          +---------+      +--------+

Features

Add, modify, delete, clone and run tasks on the web node
View real-time logs of running tasks and tasks in real time, and can terminate running tasks
Multiple task types:
- Perform http request tasks
- Run shell, python, golang、python3、nodejscodes (of course other languages can also be supported, please provide if necessary)
Father and child tasks:
When the parent task or child task is set, the parent task-> main task-> subtask will be run first. If any task fails, the entire process will be interrupted immediately. You can also set the parent task or subtask Or serial run
Scheduling Algorithm:
Supports four scheduling algorithms: random, rotation, worker weight, minimum worker number to call Worker to run tasks,
Custom alarm strategy:
Can be set to alert multiple users when the task success, failure, or run is completed
Set the return code or return content of the task to compare whether the actual return code or return content of the task is the same to determine whether the task runs successfully.
Host group:
A task can only be bound to any host group. The running of the task will select a task in this host group to run the task through the task's routing strategy
Host:
A host group can be bound to multiple hosts, the host is the node that actually runs the task, and the dispatch center automatically discovers after registration
security strategy
Certificate encryption encrypts communication data Access token
Task log management, clean up logs
Alarm notification support platform
- Mail
- WeChat
- DingDing
- Slack Channel
- Telegram Bot
- WebHook URL
Detailed task audit function
All operations on data changes to users will be recorded
Permission control
- Administrator
  Have all operating rights
- general user
  New tasks and host groups can be created, but only tasks or host groups that have been created can be operated, but audit records and all users cannot be viewed
- Guest
  Only viewing authority, no operation modification authority, but can not view audit records, all users

Supported platforms

Linux
Mac
Windows

Quick Start

git clone https://github.com/labulaka521/crocodile
cd crocodile
docker-compose up -d

Then open urlhttp://yourip:8080,you need to input a admin user and password to install it, than you will come in this system

Running

Click to download,uncompress it
installredis、mysql,then change config file core.toml
If you need to enable certificate authentication between scheduling nodes, please generate a certificate,
Run the command to generate cert certificate
```
crocodile cert
```

Then two files cert.pem and key.pem will be generated locally in the current directory. After saving these two files, fill in the path of the file in the value configuration file. Each node needs these two files

Run as a scheduler center
Multiple dispatch centers can be started to prevent single points of failure from causing dispatch to hang up
```
./crocodile server -c core.toml
```
Run as a Worker (host) node
```
/crocodile client -c core.toml
```
View version compilation information
```
./crocodile version
```
Configuration Alarm

Development

Frontend
- Install yarn
- Go to the web directory, first download the dependency yarn, and then run the front-end yarn run dev separately
- Package to go, execute make frontrnd,
Backend
- Run make runs as a schdeule center
- Run make runc as worker node

The sql directory is also packaged in go-bindata. During installation, the sql creation statement will be read from the file generated by go-bindata. If the sql is modified, re-execute make bin-data

Doc

Wiki

Donate

If this project could help you, I hope you could donate me by alipay

License

Crocodile is under the MIT license. See the LICENSE file for details.

DAXuBinbin/crocodile