Terraform deploy of Confluent Replicator for CCloud to CCloud data replication
Developed using https://github.com/confluentinc/ccloud-tools as a basis
Currently only GCP deployment available (AWS and Azure to come up, contibutions welcome ;-)
- Two Confluent Cloud Clusters Up and Running. Destination cluster is expected to run on GCP. Replicator will be deployed in same region as destination cluster
- Be Mindful that if you use VPC Peering for your cluster(s), replicator will have to be deployed in a location where it has access to both source and destination clusters. Therefore, these scripts would need to be modified to adjust networking requirements
- Outside of evaluation purposes, you need a license to use Confluent Replicator in production
- A GCP Project and Sufficient access rights to create the resources by the terraform plan
cd terraform/gcp
cp cloud.auto.tfvars.example cloud.auto.tfvars
- Edit the
cloud.auto.tfvars
file and insert your GCP project information and service account credential file (cf https://cloud.google.com/iam/docs/creating-managing-service-account-keys) cp ccloud.auto.tfvars.example ccloud.auto.tfvars
- Edit the
cloud.auto.tfvars
file and insert your Confluent Cloud Source and Destination clusters urls and credentials - Create a password file
terraform/utils/connect.password
, and insert online with your own Kafka Connect REST API user and password in this form :user:password
- File
terraform/gcp/variables.tf
: you may customize number of replicator instances to deploy, as well as, confluent version to use - File
terraform/util/kafka-replicator-config-template.json
: you may customize here some replicator configurations, like the white or black list of topics to replicate
cd terraform/gcp
terraform init
terraform plan
terraform apply
- if terraform applied successfully, you shall see a new sub-directory
load
:cd load
- You may need to wait a few minutes for the full infrastructure to be really up and then
./loadReplicator.sh
-> This script will load the replicator config through the Kafka Connect REST API
The project is licensed under the Apache 2 license.