SyncController: A Python repository from Linux Club of Peking University

Overview

This repo provides simple sync job control.

Database

Default database name: mirrors

Columns:

index: index, primary key
id: name of the mirror, also unique indentifier
state: state of the mirror, in {'error','sync','done'}
lastSyncTime: timestamp (in milliseconds) of last sync
nextSyncTime: timestamp (in milliseconds) of next sync
diskUsage: disk usage in string form
logPath: where the lastest logfile is

to create table:

CREATE TABLE `status`(
    `index` INT UNSIGNED AUTO_INCREMENT ,
    `id` VARCHAR(50) NOT NULL,
    `state` VARCHAR(20) NOT NULL,
    `lastSyncTime` BIGINT UNSIGNED DEFAULT 0,
    `nextSyncTime` BIGINT UNSIGNED DEFAULT 0,
    `diskUsage` VARCHAR(30) DEFAULT '',
    `logPath` VARCHAR(100) DEFAULT '',
    PRIMARY KEY (`index`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

RESTful API

GET /status

returns current status in JSON

an example of response:

[
    {
        "id": "manjaro",
        "state": "done",
        "lastSyncTime":1615911140966,
        "nextSyncTime":1615997400000,
        "diskUsage":"100.43G"
    }
]

CAUTION: Following APIs should be accessed by admin only.

GET /stop/$id

stop the job with id=$id

returns a JSON

// success
{"status":"ok"}

// failure
{"status":"failed","reason":"?"}

GET /start/$id

start the job with id=$id

returns a JSON

// success
{"status":"ok"}

// failure
{"status":"failed","reason":"?"}

GET /log/$id

return the job's latest log

Configuration

Configuration file is ./config.json by default. You can assign custom configuration path by adding -c <path> in command-line.

Here is an example of full configuration

{
    "mirrors": [
        {  
            "id": "manjaro",
            "schedule": "10 0 * * *",
            "provider": "rsync",
            "params": {
                "url": "rsync://mirror.yandex.ru/mirrors/manjaro/"
            },
            "metadata": {
                "name": "Manjaro",
                "url": "/manjaro/"
            },
            "file-custom": [
                "alias /abc/def/;"
            ]
        },
        {  
            "id": "opensuse",
            "schedule": "0 */8 * * *",
            "provider": "rsync",
            "params": {
                "url": "rsync://fr2.rpmfind.net/linux/opensuse/",
                "exclude": "{history,ports,source}",
                "dest": "/data-tmp/opensuse"
            }
        },
        {
            "id": "custom",
            "schedule": "10 0 * * *",
            "provider": "shell",
            "params": {
                "cmd": "node ./update_custom.js"
            }
        }
    ],
    "concurrency": 2,
    "log_dir": "/data/log/sync/",
    "repo_dir": "/data/repos/",
    "rsync": {
        "command": "rsync -avHh --delete --delete-after --delay-updates --safe-links --stats --no-o --no-g"
    },
    "mysql": {
        "host": "127.0.0.1",
        "user": "sync",
        "password": "123456",
        "database": "sync"
    },
    "port":3000
}

Attention: metadata in a mirror is used to render mirror list.

here is an example:

"metadata": {
    "url": "/ubuntu/",
    "name": "Ubuntu",
    "help": "Ubuntu",
    "describe": "A Ubuntu mirror"
}

The describe will show when your mouse hover on the link.

The help should satisfy the requirement in website/src/components/help/Ubuntu.vue currently

Be aware:

mirrors stores the list of sync jobs, in which schedule is a cron-style string indicating frequency (more info). We implemented two providers, rsync and shell, you can use them as the way shown above.
mysql stores the database connection params. See mysql for more info.
concurrency limits the max concurrency of run_queue.

How to use

Typical way:

clone the repo
modify the configuration
node index.js

You can assign configuration file by node index.js -c <path> or node index.js --conf <path>.

We also provides systemd service.

Remember to modify WorkingDirectory & ExecStart in backend.service to correct path before use

# cp backend.service /etc/systemd/system
# systemctl start backend.service

TODO