/RedisShake

redis-shake is a tool for Redis data migration and data filtering. redis-shake 是一个用于 Redis 数据迁移与过滤的工具。

Primary LanguageGoMIT LicenseMIT

redis-shake

CI

redis-shake is a tool for Redis data migration and data filtering.

Feature

  • 🚄 High performance
  • ✅ Tested on Redis 5.0, Redis 6.0 and Redis 7.0
  • 🤗 Support custom filtering rules
  • 💪 Support large instance migration
  • 💖 Support restore mode, sync mode and scan mode
  • ☁️ Support Aliyun Redis and ElastiCache

For older versions of redis-shake (support codis, twemproxy) please visit here.

redis-shake2.PNG

image.png

Document

Install

Binary package

Download from Release: https://github.com/alibaba/RedisShake/releases

Compile from source

After downloading the source code, run the sh build.sh command to compile.

git clone https://github.com/alibaba/RedisShake
cd RedisShake
sh build.sh

Usage

  1. Edit sync.toml or restore.toml.
  2. Start redis-shake.
./bin/redis-shake redis-shake.toml
# or
./bin/redis-shake restore.toml
  1. Check data synchronization status.

Configure

The redis-shake configuration file refers to sync.toml or restore.toml.

Data filtering

redis-shake supports custom filtering rules using lua scripts. redis-shake can be started with the following command:

./bin/redis-shake sync.toml filter/xxx.lua

Some following filter templates are provided in filter directory:

  1. filter/print.lua：print all commands
  2. filter/swap_db.lua：swap the data of db0 and db1

Custom filter rules

Refer to filter/print.lua to create a new lua script, and implement the filter function in the lua script. The arguments of the function are:

  • id: command id
  • is_base: is the command read from the dump.rdb file
  • group: command group, see the description file under redis/src/commands
  • cmd_name: command name
  • keys: keys in command
  • slots: slots in command
  • db_id: database id
  • timestamp_ms: timestamp of the command in milliseconds. The current version does not support it.

The return value is:

  • code
    • 0: allow this command to pass
    • 1: this command is not allowed to pass
    • 2: this command should not appear, and let redis-shake exit with an error
  • db_id: redirected db_id

Contribution

Lua script

Welcome to share more creative lua scripts.

  1. Add lua scripts under filters/.
  2. Add description to README.md.
  3. Submit a pull request.

Redis Module support

  1. Add code under internal/rdb/types.
  2. Add a command file under scripts/commands, and use the script to generate a table.go file and move it to the internal/commands directory.
  3. Add test cases under test/cases.
  4. Submit a pull request.

感谢

redis-shake 旧版是阿里云基于豌豆荚开源的 redis-port 进行二次开发的一个支持 Redis 异构集群实时同步的工具。 redis-shake v3 在 redis-shake 旧版的基础上重新组织代码结构，使其更具可维护性的版本。

redis-shake v3 参考借鉴了以下项目：