Terraform module to provision an Amazon Managed Streaming for Apache Kafka Cluster in AWS. An Amazon MSK Cluster requires a VPC to run the Broker instances in. This module provides an Internal VPC to simplify provisioning the MSK Cluster. This Internal VPC can be configured to ensure it does not collide with any existing VPCs.
By default all data is encrypted at rest using an AWS managed CMK. Users may provide their own key if they don't wish to use the AWS managed key. All data in transit is encrypted using TLS between the brokers.
A CloudWatch MSK Cluster Dashboard and CloudWatch Broker Data Log Disk Usage Alarm are optional resources available with this module. A default CloudWatch Dashboard is provided, but a custom Dashboard may also be provided. This enables clusters using enhanced monitoring to add additional metrics to the Dashboard. The CloudWatch Alarm is provided for each of the brokers in the MSK cluster to warn of Broker Disk Usage greater than 85% as per the best practices.
When using this module it is recommended that users determine the appropriate size of their MSK Cluster and understand the cost using the MSK Sizing and Pricing spreadsheet. Users should test their configurations with appropriate workloads after provisioning the cluster.
This module supports the following MSK cluster configurations:
- MSK Cluster with Default Internal VPC
- MSK Cluster with Configured Internal VPC
- MSK Cluster with an External VPC
- MSK Cluster using a Custom Kafka Broker Configuration
- MSK Cluster using Client Authentication
- MSK Cluster with CloudWatch Dashboard
- MSK Cluster with CloudWatch Broker Data Log Disk Usage Alarm
- MSK Cluster with Client Instance
These are implemented using feature flags. For information on how to configure the MSK cluster in these configurations see the examples directory. Flags can be combined, such as enabling both the CloudWatch Dashboard and the CloudWatch Broker Data Log Disk Usage Alarm.
| Name | Version |
|---|---|
| aws | ~> 2.40 |
| template | ~> 2.1.2 |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| cluster_name | Name of the MSK Cluster | string |
n/a | yes |
| broker_ebs_volume_size | Size in GiB of the EBS volume for the data drive on each broker node | number |
2000 |
no |
| broker_node_instance_type | Instance type to use for the Kafka brokers | string |
"kafka.m5.large" |
no |
| certificate_authority_arns | List of ACM Certificate Authority Amazon Resource Names (ARNS) | list(string) |
[] |
no |
| client_broker_encryption | Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS_PLAINTEXT and PLAINTEXT | string |
"TLS" |
no |
| client_subnets | A list of subnets to connect to in the client VPC | list(string) |
[] |
no |
| create_dashboard | Whether or not to create the MSK Dashboard | bool |
false |
no |
| create_diskspace_cw_alarm | Whether or not to create a Broker Diskspace CloudWatch Alarm | bool |
false |
no |
| create_msk_cluster | Whether or not to create the MSK Cluster | bool |
true |
no |
| create_vpc | Whether or not to create the MSK VPC | bool |
true |
no |
| custom_configuration_description | Description of the MSK Custom configuration | string |
"Custom MSK Configuration Example properties" |
no |
| custom_configuration_name | Name of the MSK Custom configuration | string |
"Custom-MSK-Configuration-Example" |
no |
| custom_dashboard_template | Location for the custom MSK Dashboard template | string |
"" |
no |
| encryption_kms_key_arn | KMS key short ID or ARN to use for encrypting your data at rest. If no key is specified an AWS managed KMS key will be used for encrypting the data at rest | string |
"" |
no |
| enhanced_monitoring_level | Desired enhanced MSK CloudWatch monitoring level. Valid values are DEFAULT, PER_BROKER, or PER_TOPIC_PER_BROKER | string |
"DEFAULT" |
no |
| in_cluster_encryption | Whether data communication among broker nodes is encrypted | bool |
true |
no |
| kafka_version | Desired Kafka software version | string |
"2.2.1" |
no |
| monitoring_tags | Additional tags to apply to any provisioned monitoring/metric resources | map(any) |
{} |
no |
| msk_cluster_tags | Additional tags to apply to msk_cluster resources | map(any) |
{} |
no |
| msk_configuration_arn | ARN of the MSK Configuration to use in the cluster | string |
"" |
no |
| msk_configuration_revision | Revision of the MSK Configuration to use in the cluster | number |
1 |
no |
| num_of_broker_nodes | Desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets | number |
3 |
no |
| security_groups | A list of the security groups to associate with the elastic network interfaces to control who can communicate with the cluster | list(string) |
[] |
no |
| server_properties | Contents of the server.properties file for Kafka broker | string |
"auto.create.topics.enable = false\ndefault.replication.factor = 3\ndelete.topic.enable = true\nmin.insync.replicas = 2\nnum.io.threads = 8\nnum.network.threads = 5\nnum.partitions = 1\nnum.replica.fetchers = 2\nsocket.request.max.bytes = 104857600\nunclean.leader.election.enable = true\n" |
no |
| tags | Additional tags to apply to all module resources | map(any) |
{} |
no |
| use_client_authentication | Use client authentication | bool |
false |
no |
| use_custom_configuration | Use a custom configuration on each Kafka Broker | bool |
false |
no |
| vpc_cidr_block | VPC CIDR block | string |
"10.0.0.0/16" |
no |
| vpc_id | The VPC ID for the MSK Cluster | string |
"" |
no |
| vpc_name | VPC name | string |
"MSK-VPC" |
no |
| vpc_private_subnets | Private subnets for the VPC | list(string) |
[ |
no |
| vpc_public_subnets | Public subnets for the VPC | list(string) |
[ |
no |
| vpc_tags | Additional tags to apply to any provisioned vpc resources | map(any) |
{} |
no |
| Name | Description |
|---|---|
| arn | The ARN for the MSK Cluster |
| bootstrap_brokers | List of hostname:port pairs of Kafka brokers suitable to bootstrap connectivity to the Kafka Cluster |
| bootstrap_brokers_tls | List of hostname:port pairs of Kafka brokers suitable to bootstrap connectivity to the Kafka Cluster |
| client_authentication | Certificate authority arns used for client authentication |
| cloudwatch_dashboard_arn | The ARN of the MSK Cloudwatch dashboard |
| cloudwatch_diskspace_alarm_arn | The ARN of the Broker Diskspace CloudWatch Alarm for the MSK Cluster |
| cloudwatch_diskspace_alarm_id | The ID of the Broker Diskspace CloudWatch Alarm for the MSK Cluster |
| custom_configuration_arn | Custom configuration ARN |
| custom_configuration_latest_revision | The latest revision of the MSK custom configuration |
| encryption_at_rest_kms_key_arn | The ARN of the KMS key used for encryption at rest of the broker data volume |
| msk_security_group_id | MSK Cluster Security Group ID |
| private_subnets | The private subnets in the VPC created |
| public_subnets | The public subnets in the VPC created |
| security_group | The ID of the security group created for the MSK clusters |
| vpc_id | The ID of the VPC created |
| zookeeper_connect_string | Zookeeper connection string |
Important architectural decisions along with their context and consequences are captured in Lightweight Architecture Decision Records stored in this repository. These Architecture Decision Records (ADRs) are created, updated and maintained using the ADR Tools. Instructions for installing the tools can be found here.
Please read the ADRs for this module to understand the important architectural decisions that have been made.
See LICENSE for full details.
Copyright 2020 Hypr NZ
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Copyright © 2020 Hypr NZ