/kafka-connect

Primary LanguageMustache

CP-Kafka Connect Helm Chart

This chart bootstraps a deployment of a Confluent Kafka Connect

Prerequisites

  • Kubernetes 1.9.2+
  • Helm 2.8.2+
  • A healthy and accessible Kafka Cluster

Developing Environment:

Docker Image Source:

Installing the Chart

Install along with cp-helm-charts

git clone https://github.com/confluentinc/cp-helm-charts.git
helm install cp-helm-charts

To install with a specific name, you can do:

helm install --name my-confluent cp-helm-charts

Install with a existing CP-Kafka release

helm install --set kafka.bootstrapServers="PLAINTEXT://lolling-chinchilla-cp-kafka-headless:9092",cp-schema-registry.url="lolling-chinchilla-cp-schema-registry:8081" cp-helm-charts/charts/cp-kafka-connect

Installed Components

You can use helm status <release name> to view all of the installed components.

For example:

$ helm status kissing-macaw

NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                            TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
kissing-macaw-cp-kafka-connect  ClusterIP  10.19.253.217  <none>       8083/TCP  34m

==> v1beta2/Deployment
NAME                            DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
kissing-macaw-cp-kafka-connect  1        1        1           1          34m

==> v1/Pod(related)
NAME                                             READY  STATUS   RESTARTS  AGE
kissing-macaw-cp-kafka-connect-6c77b8f5fd-cqlzq  1/1    Running  0         34m

There are

  1. A Deployment kissing-macaw-cp-kafka-connect which contains 1 Kafka Connect Pod: kissing-macaw-cp-kafka-connect-6c77b8f5fd-cqlzq.
  2. A Service kissing-macaw-cp-kafka-connect for clients to connect to Kafka Connect REST endpoint.

Configuration

You can specify each parameter using the --set key=value[,key=value] argument to helm install.

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

helm install --name my-kafka-connect -f my-values.yaml ./cp-kafka-connect

Tip: A default values.yaml is provided

Kafka Connect Deployment

The configuration parameters in this section control the resources requested and utilized by the cp-kafka-connect chart.

Parameter Description Default
replicaCount The number of Kafka Connect Servers. 1

Image

Parameter Description Default
image Docker Image of Confluent Kafka Connect. confluentinc/cp-kafka-connect
imageTag Docker Image Tag of Confluent Kafka Connect. 6.1.0
imagePullPolicy Docker Image Tag of Confluent Kafka Connect. IfNotPresent
imagePullSecrets Secrets to be used for private registries. see values.yaml for details

Port

Parameter Description Default
servicePort The port on which the Kafka Connect will be available and serving requests. 8083

Kafka Connect Worker Configurations

Parameter Description Default
configurationOverrides Kafka Connect configuration overrides in the dictionary format. {}
customEnv Custom environmental variables {}

Volumes

Parameter Description Default
volumes Volumes for connect-server container see values.yaml for details
volumeMounts Volume mounts for connect-server container see values.yaml for details

Secrets

Parameter Description Default
secrets Secret with one or more key:value pairs see values.yaml for details

Kafka Connect JVM Heap Options

Parameter Description Default
heapOptions The JVM Heap Options for Kafka Connect "-Xms512M -Xmx512M"

Resources

Parameter Description Default
resources.requests.cpu The amount of CPU to request. see values.yaml for details
resources.requests.memory The amount of memory to request. see values.yaml for details
resources.requests.limit The upper limit CPU usage for a Kafka Connect Pod. see values.yaml for details
resources.requests.limit The upper limit memory usage for a Kafka Connect Pod. see values.yaml for details

Annotations

Parameter Description Default
podAnnotations Map of custom annotations to attach to the pod spec. {}

JMX Configuration

Parameter Description Default
jmx.port The jmx port which JMX style metrics are exposed. 5555

Prometheus JMX Exporter Configuration

Parameter Description Default
prometheus.jmx.enabled Whether or not to install Prometheus JMX Exporter as a sidecar container and expose JMX metrics to Prometheus. true
prometheus.jmx.image Docker Image for Prometheus JMX Exporter container. solsson/kafka-prometheus-jmx-exporter@sha256
prometheus.jmx.imageTag Docker Image Tag for Prometheus JMX Exporter container. 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
prometheus.jmx.imagePullPolicy Docker Image Pull Policy for Prometheus JMX Exporter container. IfNotPresent
prometheus.jmx.port JMX Exporter Port which exposes metrics in Prometheus format for scraping. 5556
prometheus.jmx.resources JMX Exporter resources configuration. see values.yaml for details

Running Custom Scripts

Parameter Description Default
customEnv.CUSTOM_SCRIPT_PATH Path to external bash script to run inside the container see values.yaml for details
livenessProbe Requirement of livenessProbe depends on the custom script to be run see values.yaml for details

Deployment Topology

Parameter Description Default
nodeSelector Dictionary containing key-value-pairs to match labels on nodes. When defined pods will only be scheduled on nodes, that have each of the indicated key-value pairs as labels. Further information can be found in the Kubernetes documentation {}
tolerations Array containing taint references. When defined, pods can run on nodes, which would otherwise deny scheduling. Further information can be found in the Kubernetes documentation {}

Dependencies

Kafka

Parameter Description Default
kafka.bootstrapServers Bootstrap Servers for Kafka Connect ""