rump: A Go repository from mhoffm-aiven

Hot sync two Redis databases using dumps.

Why

There's no easy way to sync data from an AWS ElastiCache or GCP MemoryStore Redis cluster; the standard commands BGSAVE and SLAVEOF are blocked.

Rump is able to live sync Redis databases across cloud providers by only using SCAN, DUMP and RESTORE.

It's used at Sticker Mule to keep staging and development environments in sync with the production AWS/GCP Redis clusters.

Examples

# Sync local Redis DB 1 to DB 2.
$ rump -from redis://127.0.0.1:6379/1 -to redis://127.0.0.1:6379/2

# Sync ElastiCache cluster to local.
$ rump -from redis://production.cache.amazonaws.com:6379/1 -to redis://127.0.0.1:6379/1

# Sync protected ElastiCache via EC2 port forwarding.
$ ssh -L 6969:production.cache.amazonaws.com:6379 -N username@xxx.xxx.xxx.xxx &
$ rump -from redis://127.0.0.1:6969/1 -to redis://127.0.0.1:6379/1

# Dump GCP MemoryStore to file.
$ rump -from redis://10.0.20.2:6379/1 -to /backup/memorystore.rump

# Restore backup to ElastiCache.
$ rump -from /backup/memorystore.rump -to redis://production.cache.amazonaws.com:6379/1

# Sync with verbose mode disabled.
$ rump -from redis://127.0.0.1:6379/1 -to redis://127.0.0.1:6379/2 -silent

# Sync with TTLs.
$ rump -from redis://127.0.0.1:6379/1 -to redis://127.0.0.1:6379/2 -ttl

Features

Uses SCAN instead of KEYS to avoid DoS servers.
Doesn't use any temp file.
Can sync any key type.
Can optionally sync TTLs.
Uses buffered channels to optimize slow source servers.
Uses implicit pipelining to minimize network roundtrips.
Supports two-step sync: dump source to file, restore file to database.
Supports Redis URIs with auth.
Offers the same guarantees of the SCAN command.

Demo

Development

# requirements: docker, docker-compose (dc)
dc up # watch/run Rump tests and a Redis container
dc run --rm rump sh # get shell on Rump container
dc run --rm redis sh; redis-cli -h redis # get Redis console

Install

Binaries can be found on the releases page.

curl -SL https://github.com/stickermule/rump/releases/download/1.0.0/rump-1.0.0-linux-amd64 -o rump \
  && chmod +x rump;
./rump

You can run rump in a container following the Dockerfile.example.

Mentions

Maintainers

nixtrace

mhoffm-aiven/rump

Why