Charmed ZooKeeper K8s Operator

Overview

The Charmed ZooKeeper K8s Operator delivers automated operations management from day 0 to day 2 on the Apache ZooKeeper server which enables highly reliable distributed coordination, deployed on top of a Kubernetes cluster. It is an open source, end-to-end, production ready data platform on top of cloud native technologies.

This operator charm comes with features such as:

Horizontal scaling for high-availability out-of-the-box
Server-Server and Client-Server authentication both enabled by default
Access control management supported with user-provided ACL lists.

The ZooKeeper K8s Operator uses the latest upstream ZooKeeper binaries released by the The Apache Software Foundation that come with Kafka, made available using the dataplatformoci/zookeeper OCI image distributed by Canonical.

Requirements

For production environments, it is recommended to deploy at least 5 nodes for Zookeeper. While the following requirements are meant to be for production, the charm can be deployed in smaller environments.

4-8GB of RAM
2-4 cores
1 storage device, 64GB

Config options

To get a description of all config options available, please refer to the config.yaml file.

Options can be changed by using the juju config command:

juju config zookeeper-k8s <config_option_1>=<value> [<config_option_2>=<value>]

Usage

Basic usage

The ZooKeeper operator may be deployed using the Juju command line as follows:

$ juju deploy zookeeper-k8s -n 5

To watch the process, juju status can be used. Once all the units show as active|idle the credentials to access the admin user can be queried with:

juju run-action zookeeper-k8s/leader get-super-password --wait

Replication

Scaling application

The charm can be scaled using juju scale-application command.

juju scale-application zookeeper-k8s <num_of_servers_to_scale_to>

This will add or remove servers to match the required number. To scale a deployment with 3 zookeeper units to 5, run:

juju scale-application zookeeper-k8s 5

Password rotation

Internal users

The Charmed ZooKeeper K8s Operator has two internal users:

super: admin user for the cluster. Used mainly with the Kafka operator.
sync: specific to the internal quorum handling.

The set-password action can be used to rotate the password of one of them. If no username is passed, it will default to the super user.

# to set a specific password for the sync user
juju run-action zookeeper-k8s/leader set-password username=sync password=<password> --wait

# to randomly generate a password for the super user
juju run-action zookeeper-k8s/leader set-password --wait

Relations

Supported relations:

`tls-certificates` interface:

The tls-certificates interface is used with the tls-certificates-operator charm.

To enable TLS:

# deploy the TLS charm 
juju deploy tls-certificates-operator --channel=edge
# add the necessary configurations for TLS
juju config tls-certificates-operator generate-self-signed-certificates="true" ca-common-name="Test CA" 
# to enable TLS relate the application 
juju relate tls-certificates-operator zookeeper-k8s

Updates to private keys for certificate signing requests (CSR) can be made via the set-tls-private-key action.

# Updates can be done with auto-generated keys with
juju run-action zookeeper-k8s/0 set-tls-private-key --wait
juju run-action zookeeper-k8s/1 set-tls-private-key --wait
juju run-action zookeeper-k8s/2 set-tls-private-key --wait

Passing keys to internal keys should only be done with base64 -w0 not cat. With three servers this schema should be followed:

# generate shared internal key
openssl genrsa -out internal-key.pem 3072
# apply keys on each unit
juju run-action zookeeper-k8s/0 set-tls-private-key "internal-key=$(base64 -w0 internal-key.pem)"  --wait
juju run-action zookeeper-k8s/1 set-tls-private-key "internal-key=$(base64 -w0 internal-key.pem)"  --wait
juju run-action zookeeper-k8s/2 set-tls-private-key "internal-key=$(base64 -w0 internal-key.pem)"  --wait

To disable TLS remove the relation

juju remove-relation zookeeper-k8s tls-certificates-operator

Note: The TLS settings here are for self-signed-certificates which are not recommended for production clusters, the tls-certificates-operator charm offers a variety of configurations, read more on the TLS charm here

Monitoring

The Charmed ZooKeeper K8s Operator comes with several exporters by default. The metrics can be queried by accessing the following endpoints:

JMX exporter: http://<pod-ip>:9998/metrics
ZooKeeper metrics: http://<pod-ip>:7000/metrics

Additionally, the charm provides integration with the Canonical Observability Stack.

Deploy cos-lite bundle in a Kubernetes environment. This can be done by following the deployment tutorial. It is needed to offer the endpoints of the COS relations. The offers-overlay can be used, and this step is shown on the COS tutorial.

Once COS is deployed, we can find the offers from the zookeeper model:

# We are on the `cos` model. Switch to `zookeeper` model
juju switch <zookeeper_model_name>
juju find-offers <k8s_controller_name>:

A similar output should appear, if micro is the k8s controller name and cos the model where cos-lite has been deployed:

Store  URL                   Access  Interfaces                         
micro  admin/cos.grafana     admin   grafana_dashboard:grafana-dashboard
micro  admin/cos.prometheus  admin   prometheus_scrape:metrics-endpoint
. . .

Now, integrate zookeeper with the metrics-endpoint, grafana-dashboard and logging relations:

juju relate micro:admin/cos.prometheus zookeeper-k8s
juju relate micro:admin/cos.grafana zookeeper-k8s
juju relate micro:admin/cos.loki zookeeper-k8s

After this is complete, Grafana will show a new dashboard: ZooKeeper Metrics

Security

Security issues in the Charmed ZooKeeper K8s Operator can be reported through LaunchPad. Please do not file GitHub issues about security issues.

Contributing

Please see the Juju SDK docs for guidelines on enhancements to this charm following best practice guidelines, and CONTRIBUTING.md for developer guidance.

License

The Charmed ZooKeeper K8s Operator is free software, distributed under the Apache Software License, version 2.0. See LICENSE for more information.