/redis-operator

Primary LanguageGo

WARNING - WORK IN PROGRESS

WIP This project is only for learning purposes.

Intro

Redis-Cluster Operator.

Redis operator workflow

./docs/images/redis_operator.png

The redis cluster is composed by:

  • redis-cluster statefulset. The data plane.
  • Operator. A software operator that configures the cluster.

Steps:

Step 1. Configure Cluster

For each node, execute cluster meet among all IPs.

SERVICE=redis-cluster
getent hosts $SERVICE
# cartesian product IPs x IPs
redis-cli -h $IP1 -p $REDIS_PORT cluster meet $IP2 $REDIS_PORT
# sleep to allow configuration propagate
sleep $ClusterHealingTimeSeconds

Steep 2. Cluster well formed?

# for each redis node:
redis-cli -h $IP -p $PORT cluster nodes
# Get the IPs and check that are correct...
# if they are not do the meet, for all pods
redis-cli -h $IP1 -p $PORT cluster meet $IP2

Step 3. Assign Slots to Nodes

There are 16384 slots, they have to be shared among nodes.

Bash example with very slow implementation:

for slot in {0..5461}; do redis-cli -p 7000 CLUSTER ADDSLOTS $slot > /dev/null; done;
for slot in {5462..10923}; do redis-cli -p 7001 CLUSTER ADDSLOTS $slot > /dev/null; done;
for slot in {10924..16383}; do redis-cli -p 7002 CLUSTER ADDSLOTS $slot > /dev/null; done;

Better implementation:

redis-cli -p 7000 CLUSTER ADDSLOTS 0 1 2 3 4 5 ... 5461

I don’t assign slaves because I suppose --replicas = 0=, but if it is needed, note that slaves are assigned using “cluster replicate”. Example:

#  Not implemented
# redis-cli -c -p <port_of_slave> cluster replicate <node_id_of_its_master>
redis-cli -p 7003 cluster replicate 0db8943cccb9b8c3963af78019baf8d2db827f14

Golang redis libraries

Among the golang libraries that have the necessary functionalities, I compare the most used against the redis.io recommeded:

Radix pros:

  • Recommended by redis.io.
  • Better performance.
  • Minimalistic approach. You write the command text.

Example:

client, err := redis.Dial("tcp", "localhost:6379")
if err != nil {
	panic(err)
}
foo, err := client.Cmd("GET", "foo").Str()
if err != nil {
	panic(err)
}

// Checking Err field directly
err = client.Cmd("SET", "foo", "bar", "EX", 3600).Err
if err != nil {
	// handle err
}

Go-redis pros:

  • More github users and activity.
  • Exposes Redis commands as function calls. Avoiding the need to parse commands at run-time.
rdb := redis.NewClient(&redis.Options{
	Addr:     "localhost:6379",
	Password: "", // no password set
	DB:       0,  // use default DB
})

err := rdb.Set(ctx, "key", "value", 0).Err()
if err != nil {
	panic(err)
}

Election: I choose go-redis/redis because a operator does not need redis performance and simplicity andclarity in the code is important.

Redis Operator examples

Lot of examples, but that assign redis slots only this one:

Links:

Kubedb example

KubeDB uses PodDisruptionBudget to ensure that majority of these cluster replicas are available during voluntary disruptions so that quorum is maintained and no data loss is occurred.

https://kubedb.com (backup, clone)

https://kubedb.com/docs/v2020.11.12/guides/redis/

An important concept:

RedisVersion CR: https://kubedb.com/docs/v2020.11.12/guides/redis/concepts/catalog/

The redis part: https://github.com/kubedb/redis/

Operator madurity

  1. Install

Build and check

All the commands I use to build the operator.

Clean all:

OP=redis-operator
OP_KIND=Redis
OP_GROUP=redis
OPERATOR_IMAGE_REPO=redisoperator

Clean all:

echo "Warning! deleting all!"
# rm -rf $HOME/src/$OP
# OP is the operator name
mkdir -p $HOME/src/$OP
cd $HOME/src/$OP
operator-sdk init --domain=danieldorado.github.io --repo=github.com/danieldorado/$OP

Create CRD and Controller.

  • main.go initializes the Manager.
operator-sdk create api --group $OP_GROUP --version v1alpha1 --kind $OP_KIND --resource=true --controller=true

Fill scaffolds.

Define the API. Redis Types.

api/v1alpha1/redis_types.go

https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/#define-the-api

Properties:

  • Size: redis nodes number.

Invoke the controller-gen to generate *.deepcopy.go that implements runtime.Object interface:

make generate

Generate CRD manifests:

make manifests

Manifest: config/crd/bases/redis.danieldorado.github.io_redis.yaml

Use OpenAPI

Use OpenAPI to validate: https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/#openapi-validation

Define the Controller. Redis Controller.

Build and Push operator image

Kubebuilder is a prerequisite.

make docker-build docker-push IMG=danieladf/$OPERATOR_IMAGE_REPO:0.0.2

Install the CRD and deploy to the cluster the Deployment

Create and edit the role patch: config/default/role_patch.yml patch config/rbac/role.yaml

Alternatively to deploy you can run local: make run

make install
make deploy IMG=danieladf/$OPERATOR_IMAGE_REPO:0.0.2
  • Deployed in: namespace/redis-operator-system created

Create a sample.

oc project default
oc apply -f config/samples/redis_v1alpha1_redis.yaml

Clean all:

make uninstall
oc delete project/redis-operator-system

Test

Controller test:

go test ./controllers/ -v -ginkgo.v

Using envtest with ginkgo and gomega.

  • All nodes deleted at same time.

Operator Advancing topics

Conditions

import (
    "github.com/operator-framework/operator-lib/status"
)

type MyAppStatus struct {
    // Conditions represent the latest available observations of an object's state
    Conditions status.Conditions `json:"conditions"`
}

Expectations

In-memory expectations.

Guarantees.

reflect.DeepEqual

Does not work because of mutating fields added. Use a annotation hash with our fields.

hash := HashObject(expected)
expected.Annotations[ResourceHash] = hash

Other controllers

Other redis controllers that we need for the redis operator.

Examples:

  • Backup controller controller/redis_backup_controller.go
  • Restore controller controller/redis_backup_controller.go