terraform-aws-kinesis-firehose-s3-bucket: An HCL repository from Miso Robotics

Usage

Creates a Kinesis Data Firehose Delivery Stream that retrieves records from a Kinesis Data Stream and delivers them to a S3 Bucket.

module "kinesis_stream" {
  source = "dod-iac/kinesis-stream/aws"

  name = format("app-%s-%s", var.application, var.environment)
  tags = {
    Application = var.application
    Environment = var.environment
    Automation  = "Terraform"
  }
}

module "kinesis_firehose_s3_bucket" {
  source  = "dod-iac/kinesis-firehose-s3-bucket/aws"

  name = format("app-%s-firehose-%s", var.application, var.environment)

  kinesis_stream_arn = module.kinesis_stream.arn
  kinesis_role_name = format("app-%s-firehose-source-%s", var.application, var.environment)

  s3_bucket_arn = var.aws_s3_bucket_destination
  s3_role_name = format("app-%s-firehose-destination-%s", var.application, var.environment)

  tags = {
    Application = var.application
    Environment = var.environment
    Automation  = "Terraform"
  }
}

Creates a Kinesis Data Firehose Delivery Stream that retrieves records from an encrypted Kinesis Data Stream and delivers them to a S3 Bucket encrypted at-rest.

module "kinesis_kms_key" {
  source = "dod-iac/kinesis-kms-key/aws"

  name = format("alias/app-%s-kinesis-%s", var.application, var.environment)
  description = format("A KMS key used to encrypt Kinesis stream records at rest for %s:%s.", var.application, var.environment)
  tags = {
    Application = var.application
    Environment = var.environment
    Automation  = "Terraform"
  }
}

module "kinesis_stream" {
  source = "dod-iac/kinesis-stream/aws"

  name = format("app-%s-%s", var.application, var.environment)
  kms_key_id = module.kinesis_kms_key.aws_kms_key_arn
  tags = {
    Application = var.application
    Environment = var.environment
    Automation  = "Terraform"
  }
}

module "kinesis_firehose_s3_kms_key" {
  source  = "dod-iac/s3-kms-key/aws"

  name = format("alias/app-%s-firehose-destination-s3-%s", var.application, var.environment)
  description = format(
    "A KMS key used by AWS Kinesis Data Firehose Delivery Stream to encrypt objects at rest in S3 for %s:%s",
    var.application,
    var.environment
  )

  # To avoid a circular dependency, format the role ARN rather than use
  # output from the following kinesis_firehose_s3_bucket module.
  principals = [format("arn:%s:iam::%s:role/app-%s-firehose-destination-s3-%s",
    data.aws_partition.current.partition,
    data.aws_caller_identity.current.account_id,
    var.application,
    var.environment
  )]

  tags = {
    Application = var.application
    Environment = var.environment
    Automation  = "Terraform"
  }

}

module "kinesis_firehose_s3_bucket" {
  source  = "dod-iac/kinesis-firehose-s3-bucket/aws"

  name = format("app-%s-firehose-%s", var.application, var.environment)

  kinesis_stream_arn = module.kinesis_stream.arn
  kinesis_role_name = format("app-%s-firehose-source-kinesis-%s", var.application, var.environment)

  s3_bucket_arn = var.aws_s3_bucket_destination
  s3_prefix = "data/"
  s3_role_name = format("app-%s-firehose-destination-s3-%s", var.application, var.environment)
  s3_kms_key_arn = module.kinesis_firehose_s3_kms_key.aws_kms_key_arn

  tags = {
    Application = var.application
    Environment = var.environment
    Automation  = "Terraform"
  }
}

Terraform Version

Terraform 0.13. Pin module version to ~> 1.0.0 . Submit pull-requests to master branch.

Terraform 0.11 and 0.12 are not supported.

License

This project constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. However, because the project utilizes code licensed from contributors and other third parties, it therefore is licensed under the MIT License. See LICENSE file for more information.

Requirements

Name	Version
terraform	>= 1.6
aws	~> 5

Providers

Name	Version
aws	~> 5

Modules

No modules.

Resources

Name	Type
aws_iam_policy.kinesis_role	resource
aws_iam_policy.s3_role	resource
aws_iam_role.kinesis_role	resource
aws_iam_role.s3_role	resource
aws_iam_role_policy_attachment.kinesis_role	resource
aws_iam_role_policy_attachment.s3_role	resource
aws_kinesis_firehose_delivery_stream.main	resource
aws_caller_identity.current	data source
aws_iam_policy_document.assume_role_policy	data source
aws_iam_policy_document.kinesis_role	data source
aws_iam_policy_document.s3_role	data source
aws_partition.current	data source
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
cloudwatch_log_group_name	The CloudWatch Logs group name for logging. Defaults to "/aws/kinesisfirehose/[NAME]"	`string`	`""`	no
cloudwatch_log_stream_name	The CloudWatch Logs stream name for logging.	`string`	`"S3Delivery"`	no
cloudwatch_logging_enabled	Enables or disables the logging to Cloudwatch Logs.	`bool`	`false`	no
database_name	Specifies the name of the AWS Glue database that contains the schema for the output data. Required if using s3_output_data_format_conversion.	`string`	`null`	no
kinesis_role_name	The name of the AWS IAM Role for reading records from the source AWS Kinesis Stream.	`string`	n/a	yes
kinesis_role_policy_document	The contents of the IAM policy attached to the IAM role used by the Kinesis Data Firehose Delivery Stream to read records from the source AWS Kinesis Stream. If not defined, then creates a default policy.	`string`	`""`	no
kinesis_role_policy_name	The name of the IAM policy attached to the IAM Role used by the Kinesis Data Firehose Delivery Stream to read records from the source AWS Kinesis Stream. If not defined, then uses the value of the "kinesis_role_name".	`string`	`""`	no
kinesis_stream_arn	The AWS Kinesis Stream used as the source of the AWS Kinesis Data Firehose Delivery Stream.	`string`	n/a	yes
name	A name to identify the AWS Kinesis Data Firehose Delivery Stream. This is unique to the AWS account and region the stream is created in.	`string`	n/a	yes
processors	A list of processors for the AWS Kinesis Data Firehose Delivery Stream.	list(object({ type = string parameters = list(object({ key = string value = string })) }))	`[]`	no
role_arn	The role that Kinesis Data Firehose can use to access AWS Glue. This role must be in the same account you use for Kinesis Data Firehose. Cross-account roles aren't allowed. Required if using s3_output_data_format_conversion.	`string`	`null`	no
s3_bucket_arn	The ARN of the AWS S3 Bucket that receives the records.	`string`	n/a	yes
s3_buffer_interval	Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination.	`number`	`300`	no
s3_buffer_size	Buffer incoming data to the specified size, in MBs, before delivering it to the destination	`number`	`5`	no
s3_compression_format	The compression format. Options: UNCOMPRESSED, GZIP, ZIP, and Snappy.	`string`	`"UNCOMPRESSED"`	no
s3_dynamic_partitioning	If true, enable dynamic partitioning on the AWS Kinesis Data Firehose Delivery Stream.	`bool`	`false`	no
s3_dynamic_partitioning_retry_duration	Total amount of seconds Firehose spends on retries.	`number`	`300`	no
s3_error_output_prefix	Prefix added to failed records before writing them to S3. This prefix appears immediately following the bucket name.	`string`	`""`	no
s3_kms_key_arn	The ARN for the customer-managed KMS key to use for encrypt objects at rest in the AWS S3 Bucket.	`string`	`""`	no
s3_output_data_format_conversion	Convert the data to the specified format before writing to S3.	`string`	`null`	no
s3_prefix	An extra S3 Key prefix prepended before the time format prefix of records delivered to the AWS S3 Bucket.	`string`	`""`	no
s3_role_name	The name of the AWS IAM Role for delivering files to the destination AWS S3 Bucket.	`string`	n/a	yes
s3_role_policy_document	The contents of the IAM policy attached to the IAM role used by the Kinesis Data Firehose Delivery Stream for delivering data to the AWS S3 Bucket. If not defined, then creates the policy based on allowed actions.	`string`	`""`	no
s3_role_policy_name	The name of the IAM policy attached to the IAM Role used by the Kinesis Data Firehose Delivery Stream. If not defined, then uses the value of the "s3_role_name".	`string`	`""`	no
table_name	Specifies the AWS Glue table that contains the column information that constitutes your data schema. Required if using s3_output_data_format_conversion.	`string`	`null`	no
tags	Tags applied to the AWS Kinesis Data Firehose Delivery Stream.	`map(string)`	`{}`	no

Outputs

Name	Description
kinesis_firehose_delivery_stream_arn	The ARN of the Kinesis Data Firehose Delivery Stream
kinesis_firehose_delivery_stream_name	The name of the Kinesis Data Firehose Delivery Stream

MisoRobotics/terraform-aws-kinesis-firehose-s3-bucket