Dockerfile for Kafka Connect, based on dongjinleekr/kafka-docker and in turn its upstream, wurstmeister/kafka-docker.
This image includes the following connect plugins and converters:
plugins:
- kafka-connect-jdbc (with mysql-connector-java)
- kafka-connect-hdfs
converters:
- avro-converter
- json-schema-converter
- protobuf-converter
version | based on | mysql-connector-java | converters | plugins |
---|---|---|---|---|
2.13-2.5.1 | Kafka 2.13-2.5.1 | 8.0.22 | 5.5.2 | 5.5.2 |
The following configuration shows how to configure a kafka connect cluster with this Docker image in Kubernetes cluster, with a Kafka cluster available in djlee-kafka-headless
service.
Note:
- For configuring Kafka cluster, see here.
- For configuring Schema Registry cluster, see here.
- This configuration is intended for dev or testing purpose; it may be used in production environment, but I can't give any guarantees in that respect.
apiVersion: apps/v1
kind: Deployment
metadata:
name: djlee-connect
labels:
app: djlee-connect
spec:
replicas: 3
selector:
matchLabels:
app: djlee-connect
template:
metadata:
labels:
app: djlee-connect
spec:
containers:
- name: djlee-connect
image: dongjinleekr/kafka-connect:2.13-2.5.1
imagePullPolicy: IfNotPresent
ports:
- name: kafka-connect
containerPort: 8083
protocol: TCP
- name: jmx
containerPort: 9999
env:
- name: CONNECT_BOOTSTRAP_SERVERS
value: "djlee-kafka-0.djlee-kafka-headless:9092,djlee-kafka-1.djlee-kafka-headless:9092,djlee-kafka-2.djlee-kafka-headless:9092,djlee-kafka-3.djlee-kafka-headless:9092"
- name: CONNECT_GROUP_ID
value: "djlee-connect"
- name: CONNECT_CONFIG_STORAGE_TOPIC
value: "connect-config"
- name: CONNECT_OFFSET_STORAGE_TOPIC
value: "connect-offsets"
- name: CONNECT_STATUS_STORAGE_TOPIC
value: "connect-status"
- name: CONNECT_KEY_CONVERTER
value: "io.confluent.connect.avro.AvroConverter"
- name: CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL
value: "djlee-schema-registry-service"
- name: CONNECT_VALUE_CONVERTER
value: "io.confluent.connect.avro.AvroConverter"
- name: CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL
value: "djlee-schema-registry-service"
- name: CONNECT_REST_ADVERTISED_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CONNECT_JMX_PORT
value: "9999"
---
apiVersion: v1
kind: Service
metadata:
name: djlee-kafka-connect-service
labels:
app: djlee-connect
spec:
ports:
- name: kafka-connect
port: 8083
selector:
app: djlee-connect
As you can see above, a environment variale named with CONNECT_A_B
corresponds to a configuration property of a.b
. For example, CONNECT_KEY_CONVERTER
corresponds to [key.converter
property](](https://kafka.apache.org/documentation/#connectconfigs_key.converter) in Kafka Connect configuration).
As described above, this image includes some connector plugins by default. If you need a image with additional plugins, create a Dockerfile like below and build a custom extension image.
FROM dongjinleekr/kafka-connect:latest
# before building the image, place built connector plugins in converters directory.
COPY converters /usr/local/share/java/converters
To build the image yourself, run following:
# Build image based on dongjinleekr/kafka:2.13-2.7.0 (Scala 2.13, Apache Kafka 2.7.0)
# before building the image, place built connectors and converters into the connectors/ and converters/ directory, respectively.
SCALA_VERSION=2.13 KAFKA_VERSION=2.7.0 && docker build --build-arg scala_version=${SCALA_VERSION} --build-arg kafka_version=${KAFKA_VERSION} -t dongjinleekr/kafka-connect:${SCALA_VERSION}-${KAFKA_VERSION} .