/terraform-kubernetes-confluent-platform

A Terraform module for managing Confluent for Kubernetes (CFK).

Primary LanguageGoApache License 2.0Apache-2.0

Pre-Commit cookiecutter-tf-module Terratest

terraform-kubernetes-confluent-platform

A Terraform module for managing Confluent for Kubernetes (CFK).

Prerequisites

It is recommended that you at least install Custom Resource Definitions (CRDs) on the Kubernetes cluster before applying the Terraform.

Please see The Confluent for Kubernetes Quickstart for more information.

Override CFK Manifest Values

Similar to the values file for Helm; The variables supplied to the Confluent Platform module will be deep-merged with the default local values of the module.

Example

module "confluent_platform" {
  source  = "aidanmelen/confluent-platform/kubernetes"
  version = ">= 0.9.5"

  namespace = var.namespace

  # assumes the confluent operator was deployed in another terraform run
  confluent_operator = {
    create = false
  }

  # uncomment to override the modules default local values
  /*
  zookeeper      = yamldecode(file("${path.module}/values/zookeeper.yaml"))
  kafka          = yamldecode(file("${path.module}/values/kafka.yaml"))
  connect        = yamldecode(file("${path.module}/values/connect.yaml"))
  ksqldb         = yamldecode(file("${path.module}/values/ksqldb.yaml"))
  controlcenter  = yamldecode(file("${path.module}/values/controlcenter.yaml"))
  schemaregistry = yamldecode(file("${path.module}/values/schemaregistry.yaml"))
  kafkarestproxy = yamldecode(file("${path.module}/values/kafkarestproxy.yaml"))
  */

  create_controlcenter = var.create_controlcenter
}

Usage

Release the Confluent Platform with:

cd examples/complete
terraform init
terraform apply

Troubleshooting

Error: Failed to determine GroupVersionResource for manifest

This Terraform module uses the kubernetes_manifest resource to deploy CFK custom resources. The following excerpt is from Beta Support for CRDs in the Terraform Provider for Kubernetes:

Custom resource definitions must be applied before custom resources. As above, this is because the provider queries the Kubernetes API for the OpenAPI specification for the resource supplied in the manifest attribute. If the CRD doesn’t exist in the OpenAPI specification during plan time then Terraform can’t use it to create custom resources.

Please see Troubleshoot Confluent for Kubernetes for other troubleshooting needs.

Tests

Run Terratest using the Makefile targets:

  1. make setup
  2. make tests

Results

Terratest Suite (Module v0.9.5) (Terraform v1.2.6)
--- PASS: TestTerraformCompleteExample (175.48s)
--- PASS: TestTerraformConfluentOperatorExample (22.24s)
--- PASS: TestTerraformConfluentPlatformExample (236.10s)
--- PASS: TestTerraformConfluentPlatformSinglenodeExample (232.32s)
--- PASS: TestTerraformConnectorExample (222.19s)
--- PASS: TestTerraformKafkaTopicsExample (168.37s)
--- PASS: TestTerraformSchemaExample (336.87s)

Makefile Targets

help                                This help.
build                               Build docker dev container
run                                 Run docker dev container
setup                               Setup project
lint                                Lint with pre-commit
lint-all                            Lint all files with pre-commit
tests                               Tests with Terratest
test-confluent-operator             Test the confluent_operator example
test-confluent-platform             Test the confluent_platform example
test-confluent-platform-singlenode  Test the confluent_platform_singlenode example
test-complete                       Test the complete example
test-kafka-topic                    Test the kafka_topic example
test-schema                         Test the schema example
test-connector                      Test the connector example
release                             Tag remote triggering Terraform Registry release
clean                               Clean project

Requirements

Name Version
terraform >= 0.14.8
helm >= 2.0.0
kubernetes >= 2.12.1

Providers

Name Version
kubernetes >= 2.12.1

Modules

Name Source Version
confluent_operator ./modules/confluent_operator n/a
confluent_platform_override_values Invicton-Labs/deepmerge/null 0.1.5
confluent_role_bindings ./modules/confluent_role_binding n/a
connectors ./modules/connector n/a
kafka_rest_classes ./modules/kafka_rest_class n/a
kafka_topics ./modules/kafka_topic n/a
schemas ./modules/schema n/a

Resources

Name Type
kubernetes_manifest.components resource
kubernetes_resource.components data source

Inputs

Name Description Type Default Required
confluent_operator Controls if the Confluent Operator resources should be created. This is required when the Confluent Operator is not already running on the kubernetes cluster. any 
{
  "create": true
}
 no
confluent_operator_app_version The default Confluent Operator app version. This may be overriden by component override values. This version must be compatible with the confluent_platform_version. Please see confluent docs for more information: https://docs.confluent.io/platform/current/installation/versions-interoperability.html#confluent-operator string "2.4.0" no
confluent_platform_version The default Confluent Platform app version. This may be overriden by component override values. This version must be compatible with the confluent_operator_app_version. Please see confluent docs for more information: https://docs.confluent.io/platform/current/installation/versions-interoperability.html#confluent-operator string "7.2.0" no
confluent_role_bindings A map of Confluent Role Bindings to create. The key is the confluent role binding name. The value map is the input for the confluent_role_binding submodule. any {} no
connect The Connect override values. any {} no
connectors A map of Connectors to create. The key is the connector name. The value map is the input for the connector submodule. any {} no
controlcenter The ControlCenter override values. any {} no
create Controls if the Confluent Platform and Operator resources should be created (affects all resources). bool true no
create_connect Controls if the Connect component of the Confluent Platform should be created. bool true no
create_controlcenter Controls if the ControlCenter component of the Confluent Platform should be created. bool true no
create_kafka Controls if the Kafka component of the Confluent Platform should be created. bool true no
create_kafkarestproxy Controls if the KafkaRestProxy component of the Confluent Platform should be created. bool true no
create_ksqldb Controls if the KsqlDB component of the Confluent Platform should be created. bool true no
create_schemaregistry Controls if the SchemaRegistry component of the Confluent Platform should be created. bool true no
create_timeout The create timeout for each Confluent Platform component. string "1h" no
create_zookeeper Controls if the Zookeeper component of the Confluent Platform should be created. bool true no
delete_timeout The delete timeout for each Confluent Platform component. string "10m" no
kafka The Kafka override values. any {} no
kafka_rest_classes A map of Kafka Rest Classes to create. The key is the kafka rest class name. The value map is the input for the kafka_rest_class submodule. any {} no
kafka_topics A map of Kafka Topics to create. The key is the topic name. The value map is the input for the kafka_topic submodule. any {} no
kafkarestproxy The KafkaRestProxy override values. any {} no
ksqldb The KsqlDB override values. any {} no
namespace The namespace to release Confluent Platform into. When confluent_operator is specified, this will also ensure the Confluent Operator is released into the same namespace. string "confluent" no
schemaregistry The SchemaRegistry override values. any {} no
schemas A map of Schemas to create. The key is the schema name. The value map is the input for the schema submodule. any {} no
update_timeout The update timeout for each Confluent Platform component. string "1h" no
zookeeper The Zookeeper override values. any {} no

Outputs

Name Description
confluent_operator Map of attributes for the Confluent Operator.
confluent_role_binding_manifests Map of attribute maps for all the ConfluentRoleBinding manifests created.
confluent_role_binding_objects Map of attribute maps for all the ConfluentRoleBinding objects created.
confluent_role_bindings Map of attribute maps for all ConfluentRoleBinding submodules created.
connect_manifest The Connect manifest.
connect_object The Connect object.
connector_manifests Map of attribute maps for all the Connector manifests created.
connector_objects Map of attribute maps for all the Connector objects created.
connectors Map of attribute maps for all Connector submodules created.
controlcenter_manifest The ControlCenter manifest.
controlcenter_object The ControlCenter object.
kafka_manifest The Kafka manifest.
kafka_object The Kafka object.
kafka_rest_class_manifests Map of attribute maps for all the KafkaRestClass manifests created.
kafka_rest_class_objects Map of attribute maps for all the KafkaRestClass objects created.
kafka_rest_classes Map of attribute maps for all KafkaRestClass submodules created.
kafka_topic_manifests Map of attribute maps for all the KafkaTopic manifests created.
kafka_topic_objects Map of attribute maps for all the KafkaTopic objects created.
kafka_topics Map of attribute maps for all KafkaTopic submodules created.
kafkarestproxy_manifest The KafkaRestProxy manifest.
kafkarestproxy_object The KafkaRestProxy object.
ksqldb_manifest The KsqlDB manifest.
ksqldb_object The KsqlDB object.
namespace The default namespace for the Confluent Platform.
schema_config_map Map of attribute maps for all the Schema ConfigMap created.
schema_manifests Map of attribute maps for all the Schema manifests created.
schema_objects Map of attribute maps for all the Schema objects created.
schemaregistry_manifest The SchemaRegistry manifest.
schemaregistry_object The SchemaRegistry object.
schemas Map of attribute maps for all Schema submodules created.
zookeeper_manifest The Zookeeper manifest.
zookeeper_object The Zookeeper object.

License

Apache 2 Licensed. See LICENSE for full details.