/terraform-aws-msk-module

Primary LanguageHCLApache License 2.0Apache-2.0

Amazon MSK Terraform Module

Terraform module to provision an Amazon Managed Streaming for Apache Kafka Cluster in AWS. An Amazon MSK Cluster requires a VPC to run the Broker instances in. This module provides an Internal VPC to simplify provisioning the MSK Cluster. This Internal VPC can be configured to ensure it does not collide with any existing VPCs.

By default all data is encrypted at rest using an AWS managed CMK. Users may provide their own key if they don't wish to use the AWS managed key. All data in transit is encrypted using TLS between the brokers.

A CloudWatch MSK Cluster Dashboard and CloudWatch Broker Data Log Disk Usage Alarm are optional resources available with this module. A default CloudWatch Dashboard is provided, but a custom Dashboard may also be provided. This enables clusters using enhanced monitoring to add additional metrics to the Dashboard. The CloudWatch Alarm is provided for each of the brokers in the MSK cluster to warn of Broker Disk Usage greater than 85% as per the best practices.

Good Practices

When using this module it is recommended that users determine the appropriate size of their MSK Cluster and understand the cost using the MSK Sizing and Pricing spreadsheet. Users should test their configurations with appropriate workloads after provisioning the cluster.

Features & Examples

This module supports the following MSK cluster configurations:

  1. MSK Cluster with Default Internal VPC
  2. MSK Cluster with Configured Internal VPC
  3. MSK Cluster with an External VPC
  4. MSK Cluster using a Custom Kafka Broker Configuration
  5. MSK Cluster using Client Authentication
  6. MSK Cluster with CloudWatch Dashboard
  7. MSK Cluster with CloudWatch Broker Data Log Disk Usage Alarm
  8. MSK Cluster with Client Instance

These are implemented using feature flags. For information on how to configure the MSK cluster in these configurations see the examples directory. Flags can be combined, such as enabling both the CloudWatch Dashboard and the CloudWatch Broker Data Log Disk Usage Alarm.

Providers

Name Version
aws ~> 2.40
template ~> 2.1.2

Inputs

Name Description Type Default Required
cluster_name Name of the MSK Cluster string n/a yes
broker_ebs_volume_size Size in GiB of the EBS volume for the data drive on each broker node number 2000 no
broker_node_instance_type Instance type to use for the Kafka brokers string "kafka.m5.large" no
certificate_authority_arns List of ACM Certificate Authority Amazon Resource Names (ARNS) list(string) [] no
client_broker_encryption Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS_PLAINTEXT and PLAINTEXT string "TLS" no
client_subnets A list of subnets to connect to in the client VPC list(string) [] no
create_dashboard Whether or not to create the MSK Dashboard bool false no
create_diskspace_cw_alarm Whether or not to create a Broker Diskspace CloudWatch Alarm bool false no
create_msk_cluster Whether or not to create the MSK Cluster bool true no
create_vpc Whether or not to create the MSK VPC bool true no
custom_configuration_description Description of the MSK Custom configuration string "Custom MSK Configuration Example properties" no
custom_configuration_name Name of the MSK Custom configuration string "Custom-MSK-Configuration-Example" no
custom_dashboard_template Location for the custom MSK Dashboard template string "" no
encryption_kms_key_arn KMS key short ID or ARN to use for encrypting your data at rest. If no key is specified an AWS managed KMS key will be used for encrypting the data at rest string "" no
enhanced_monitoring_level Desired enhanced MSK CloudWatch monitoring level. Valid values are DEFAULT, PER_BROKER, or PER_TOPIC_PER_BROKER string "DEFAULT" no
in_cluster_encryption Whether data communication among broker nodes is encrypted bool true no
kafka_version Desired Kafka software version string "2.2.1" no
monitoring_tags Additional tags to apply to any provisioned monitoring/metric resources map(any) {} no
msk_cluster_tags Additional tags to apply to msk_cluster resources map(any) {} no
msk_configuration_arn ARN of the MSK Configuration to use in the cluster string "" no
msk_configuration_revision Revision of the MSK Configuration to use in the cluster number 1 no
num_of_broker_nodes Desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets number 3 no
security_groups A list of the security groups to associate with the elastic network interfaces to control who can communicate with the cluster list(string) [] no
server_properties Contents of the server.properties file for Kafka broker string "auto.create.topics.enable = false\ndefault.replication.factor = 3\ndelete.topic.enable = true\nmin.insync.replicas = 2\nnum.io.threads = 8\nnum.network.threads = 5\nnum.partitions = 1\nnum.replica.fetchers = 2\nsocket.request.max.bytes = 104857600\nunclean.leader.election.enable = true\n" no
tags Additional tags to apply to all module resources map(any) {} no
use_client_authentication Use client authentication bool false no
use_custom_configuration Use a custom configuration on each Kafka Broker bool false no
vpc_cidr_block VPC CIDR block string "10.0.0.0/16" no
vpc_id The VPC ID for the MSK Cluster string "" no
vpc_name VPC name string "MSK-VPC" no
vpc_private_subnets Private subnets for the VPC list(string)
[
"10.0.1.0/24",
"10.0.2.0/24",
"10.0.3.0/24"
]
no
vpc_public_subnets Public subnets for the VPC list(string)
[
"10.0.0.0/24"
]
no
vpc_tags Additional tags to apply to any provisioned vpc resources map(any) {} no

Outputs

Name Description
arn The ARN for the MSK Cluster
bootstrap_brokers List of hostname:port pairs of Kafka brokers suitable to bootstrap connectivity to the Kafka Cluster
bootstrap_brokers_tls List of hostname:port pairs of Kafka brokers suitable to bootstrap connectivity to the Kafka Cluster
client_authentication Certificate authority arns used for client authentication
cloudwatch_dashboard_arn The ARN of the MSK Cloudwatch dashboard
cloudwatch_diskspace_alarm_arn The ARN of the Broker Diskspace CloudWatch Alarm for the MSK Cluster
cloudwatch_diskspace_alarm_id The ID of the Broker Diskspace CloudWatch Alarm for the MSK Cluster
custom_configuration_arn Custom configuration ARN
custom_configuration_latest_revision The latest revision of the MSK custom configuration
encryption_at_rest_kms_key_arn The ARN of the KMS key used for encryption at rest of the broker data volume
msk_security_group_id MSK Cluster Security Group ID
private_subnets The private subnets in the VPC created
public_subnets The public subnets in the VPC created
security_group The ID of the security group created for the MSK clusters
vpc_id The ID of the VPC created
zookeeper_connect_string Zookeeper connection string

Architectural Decision Records

Important architectural decisions along with their context and consequences are captured in Lightweight Architecture Decision Records stored in this repository. These Architecture Decision Records (ADRs) are created, updated and maintained using the ADR Tools. Instructions for installing the tools can be found here.

Please read the ADRs for this module to understand the important architectural decisions that have been made.

License

License

See LICENSE for full details.

Copyright 2020 Hypr NZ

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Copyright © 2020 Hypr NZ