debezium-oracle-kubernetes-without-operator
Debezium is CDC(change data capture) tool mainly used for replication or streaming to catch data change in row level for most of RDBMS. Using operator to deploy into microservices sometimes too much with much features we not used actually Strimzi for example. So here we are using kubernetes manifests for the sake of simplicity.
Prerequisite
- Installed kubernetes and docker
Here i used docker desktop with enabling kubernetes cluster by Preferences -> Kubernetes -> Enable Kubernetes -> Apply & restart
You also can using minikube, kubespray or native kubernetes(latest kubernetes version no longger support docker as container runtime only left cri-o or containerd as container runtime) - Download oracle client
You can find it here instant-client - Oracle prep
You can find how to prepare oracle database here Oracle-Prep
Getting Started
First thing we have to setup local registry for docker image we built
docker run -d -p 5000:5000 --restart=always --name registry -e REGISTRY_VALIDATION_DISABLED=true registry:2
Then build our debezium custom image
docker build -f Dockerfile -t localhost:5000/debezium_custom:latest .
Push image to local registry
docker push localhost:5000/debezium_custom:latest
Create namespace big-data for this project
kubectl create namespace big-data
Now we ready to deploy it to kubernetes
kubectl create -f debezium.yaml
Let see our pods
kubectl get po -n big-data
Create debezium connector
kubectl exec -it debezium-connect-7d45648fbb-rnmml -n big-data -- curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" debezium-connect-7d45648fbb-rnmml:8083/connectors/ -d @- <<EOF
{
"name": "test-connector",
"config": {
"connector.class": "io.debezium.connector.oracle.OracleConnector",
"tasks.max": "1",
"database.server.name": "xxxxx",
"database.oracle.version": 11,
"database.hostname": "xxxxx",
"database.port": "1521",
"database.user": "xxxxx",
"database.password": "xxxxx",
"database.dbname" : "orcltes",
"table.include.list": "2000bgr.test",
"database.out.server.name": "dboprxout",
"include.schema.changes": "true",
"snapshot.mode": "initial",
"topic.prefix": "dbz-topic",
"database.tablename.case.insensitive": "true",
"database.history.skip.unparseable.ddl": "true",
"errors.log.enable": "true",
"database.history.kafka.topic": "dbz-history-topic",
"database.history.kafka.bootstrap.servers": "debezium-kafka:9092",
"debezium.log.mining.transaction.retention.hours": "1",
"event.processing.failure.handing.mode": "warn",
"log.mining.batch.size.default": "100000",
"log.mining.batch.size.min": "10000",
"log.mining.batch.size.max": "1000000",
"log.mining.sleep.time.default": "200",
"log.mining.sleep.time.min": "0",
"log.mining.sleep.time.max": "1000",
"log.mining.sleep.time.increment": "100",
"log.mining.scn.gap.detection.gap.size.min": "1000"
}
}
EOF
Let see our kafka topic
kubectl run kafka-topics -it -n big-data --image=debezium/kafka:latest --rm=true --restart=Never -- bin/kafka-topics.sh --list --bootstrap-server debezium-kafka:9092
Finished. Now we can consume data from kafka
kubectl run kafka-consumer -it -n big-data --image=debezium/kafka:latest --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server debezium-kafka:9092 --topic xxxxx --from-beginning