debezium-oracle-kubernetes-without-operator

Debezium is CDC(change data capture) tool mainly used for replication or streaming to catch data change in row level for most of RDBMS. Using operator to deploy into microservices sometimes too much with much features we not used actually Strimzi for example. So here we are using kubernetes manifests for the sake of simplicity.

Prerequisite

Installed kubernetes and docker
Here i used docker desktop with enabling kubernetes cluster by Preferences -> Kubernetes -> Enable Kubernetes -> Apply & restart
You also can using minikube, kubespray or native kubernetes(latest kubernetes version no longger support docker as container runtime only left cri-o or containerd as container runtime)
Download oracle client
You can find it here instant-client
Oracle prep
You can find how to prepare oracle database here Oracle-Prep

Getting Started

First thing we have to setup local registry for docker image we built

docker run -d -p 5000:5000 --restart=always --name registry -e REGISTRY_VALIDATION_DISABLED=true registry:2

Then build our debezium custom image

docker build -f Dockerfile -t localhost:5000/debezium_custom:latest .

Push image to local registry

docker push localhost:5000/debezium_custom:latest

Create namespace big-data for this project

kubectl create namespace big-data

Now we ready to deploy it to kubernetes

kubectl create -f debezium.yaml

Let see our pods

kubectl get po -n big-data

Create debezium connector

kubectl exec -it debezium-connect-7d45648fbb-rnmml -n big-data -- curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" debezium-connect-7d45648fbb-rnmml:8083/connectors/ -d @- <<EOF
{
    "name": "test-connector",
    "config": {
        "connector.class": "io.debezium.connector.oracle.OracleConnector",
        "tasks.max": "1",
        "database.server.name": "xxxxx",
        "database.oracle.version": 11,
        "database.hostname": "xxxxx",
        "database.port": "1521",
        "database.user": "xxxxx",
        "database.password": "xxxxx",
        "database.dbname" : "orcltes",
        "table.include.list": "2000bgr.test",
        "database.out.server.name": "dboprxout",
        "include.schema.changes": "true",
        "snapshot.mode": "initial",
        "topic.prefix": "dbz-topic",
        "database.tablename.case.insensitive": "true",
        "database.history.skip.unparseable.ddl": "true",
        "errors.log.enable": "true",
        "database.history.kafka.topic": "dbz-history-topic",
        "database.history.kafka.bootstrap.servers": "debezium-kafka:9092",
        "debezium.log.mining.transaction.retention.hours": "1",
        "event.processing.failure.handing.mode": "warn",
        "log.mining.batch.size.default": "100000",
        "log.mining.batch.size.min": "10000",
        "log.mining.batch.size.max": "1000000",
        "log.mining.sleep.time.default": "200",
        "log.mining.sleep.time.min": "0",
        "log.mining.sleep.time.max": "1000",
        "log.mining.sleep.time.increment": "100",
        "log.mining.scn.gap.detection.gap.size.min": "1000"
    }
}
EOF

Let see our kafka topic

kubectl run kafka-topics -it -n big-data --image=debezium/kafka:latest --rm=true --restart=Never -- bin/kafka-topics.sh --list --bootstrap-server debezium-kafka:9092

Finished. Now we can consume data from kafka

kubectl run kafka-consumer -it -n big-data --image=debezium/kafka:latest --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server debezium-kafka:9092 --topic xxxxx --from-beginning

dguimard/debezium-oracle-kubernetes-without-operator

debezium-oracle-kubernetes-without-operator

Prerequisite

Getting Started