/terraform-aws-collection

Terraform module which streamlines collection from multiple sources within AWS

Primary LanguageHCLOtherNOASSERTION

Observe AWS Collection

This module assembles different methods of collecting data from AWS into Observe. It is intended as both a starting point and as a reference.

The module sets up the following forwarding methods:

  • the Observe lambda
  • an S3 bucket, subscribed to the aforementioned Lambda
  • a Firehose stream

Given these egresses, we extract data from the following sources:

  • Cloudwatch Metrics, via Firehose
  • CloudTrail, via S3
  • EventBridge, via Firehose
  • AWS snapshot data, via Lambda

Usage

The following snippet installs the Observe AWS collection stack to a single region:

module "observe_collection" {
  source           = "github.com/observeinc/terraform-aws-collection"
  observe_customer = ""
  observe_token    = ""
}

Common Options

The snippet below installs the Observe AWS collection stack so that all supported CloudWatch Logs, CloudWatch metrics, CloudTrail records, and AWS resource updates are collected, except for some excluded items:

module "observe_collection" {
  source           = "github.com/observeinc/terraform-aws-collection"
  observe_customer = ""
  observe_token    = ""
  
  subscribed_log_group_matches = [".*"]
  subscribed_log_group_excludes = ["/aws/elasticbeanstalk/my-app.*"]
  snapshot_exclude = ["kms:Describe*"]
  cloudwatch_metrics_exclude_filters = ["AWS/KMS"]
}

Diagram

 ┌──────────────────┐                          ┌───────────────┐    ┌─────────────┐
 │cloudwatch metrics├──┐                       │   s3 bucket   │    │  cloudtrail │
 └──────────────────┘  │           ┌───────────►               ◄────┤             │
                       │           │           └────────┬──────┘    └─────────────┘
                       │           │                    │
                       │           │                    │
                       │     ┌─────┴──────┐             │
                       └─────►            │             │
                             │  Firehose  ├──────┐      │
         ┌───────────────────►            │      │      │
         │                   └───▲──┬─────┘      │      │
         │                       │  │            │      │
         │                       │  │        ┌───▼───┐  │
   ┌─────┴─────┐                 │  │        │       │  │
   │eventbridge│                 │  │        │observe│  │
   └─────┬─────┘                 │  │        │       │  │
         │            ┌──────────┴──▼─┐      └────▲──┘  │
         │            │cloudwatch logs│           │     │
         │            └──────────┬──┬─┘           │     │
         │                       │  │             │     │
         │                       │  │             │     │
         │                   ┌───┴──▼─────┐       │     │
         └───────────────────►            ├───────┘     │
                             │   Lambda   │             │
         ┌───────────────────►            ◄─────────────┘
         │                   └────────────┘
┌────────┴─────────┐
│ cloudwatch logs  │
└──────────────────┘

Requirements

Name Version
terraform >= 1.2
aws >= 5.0
random >= 3.0.0

Providers

Name Version
aws >= 5.0
random >= 3.0.0

Modules

Name Source Version
lambda_log_subscription observeinc/kinesis-firehose/aws//modules/cloudwatch_logs_subscription 2.2.0
observe_cloudwatch_logs_subscription observeinc/cloudwatch-logs-subscription/aws 0.5.0
observe_cloudwatch_metrics observeinc/kinesis-firehose/aws//modules/cloudwatch_metrics 2.2.0
observe_firehose_eventbridge observeinc/kinesis-firehose/aws//modules/eventbridge 2.2.0
observe_kinesis_firehose observeinc/kinesis-firehose/aws 2.2.0
observe_lambda observeinc/lambda/aws 3.5.1
observe_lambda_s3_bucket_subscription observeinc/lambda/aws//modules/s3_bucket_subscription 3.5.1
observe_lambda_snapshot observeinc/lambda/aws//modules/snapshot 3.5.1
s3_bucket terraform-aws-modules/s3-bucket/aws ~> 4.0

Resources

Name Type
aws_cloudtrail.trail resource
aws_cloudwatch_event_rule.rules resource
aws_cloudwatch_log_group.group resource
random_string.this resource
aws_caller_identity.current data source
aws_iam_policy_document.bucket data source
aws_region.current data source

Inputs

Name Description Type Default Required
cloudtrail_enable Whether to create a CloudTrail trail.

Useful for avoiding the 'trails per region' quota of 5, such as when testing.		 bool true no
cloudtrail_enable_log_file_validation Whether log file integrity validation is enabled for CloudTrail. Defalults to false. bool false no
cloudtrail_exclude_management_event_sources A list of management event sources to exclude.

See the following link for more info:
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-events-with-cloudtrail.html		 set(string) 
[
  "kms.amazonaws.com",
  "rdsdata.amazonaws.com"
]
 no
cloudtrail_is_multi_region_trail Whether to enable multi region trail export bool true no
cloudwatch_metrics_exclude_filters Namespaces to exclude. Mutually exclusive with cloudwatch_metrics_include_filters.

To disable Cloudwatch Metrics Stream entirely, use ["*"].		 set(string) [] no
cloudwatch_metrics_include_filters Namespaces to include. Mutually exclusive with cloudwatch_metrics_exclude_filters. set(string) [] no
dead_letter_queue_destination Send failed events/function executions to a dead letter queue arn sns or sqs string null no
enable_s3_bucket_eventbridge Enable sending bucket notifications to EventBridge bool false no
eventbridge_rules Eventbridge events matching these rules will be forwarded to Observe. Map
keys are only used to provide stable resource addresses.

If null, a default set of rules will be used.		 
map(object({
    description   = string
    event_pattern = string
  }))
 null no
invoke_snapshot_on_start_enabled Toggle invocation of snapshot from Cloudformation. This can be useful for debug purposes if the lambda fails to complete successfully. bool false no
kms_key_id KMS key ARN to use to encrypt the logs delivered by CloudTrail. string "" no
lambda_envvars Environment variables map(any) {} no
lambda_memory_size The amount of memory that your function has access to. Increasing the function's memory also increases its CPU allocation.
The default value is 256 MB. The value must be a multiple of 64 MB.		 number 256 no
lambda_reserved_concurrent_executions The number of simultaneous executions to reserve for the function. number 100 no
lambda_s3_custom_rules List of rules to evaluate how to upload a given S3 object to Observe. 
list(object({
    pattern = string
    headers = map(string)
  }))
 [] no
lambda_subscribe_logs Whether to subscribe to the Lambda function's logs and deliver them from CloudWatch to Observe via Kinesis Firehose. bool true no
lambda_timeout The amount of time that Lambda allows a function to run before stopping it.
The maximum allowed value is 900 seconds.		 number 120 no
lambda_version Lambda version string "arm64/latest" no
log_subscription_name Name for log subscription resources to be created string null no
name Name for resources to be created string "observe-collection" no
observe_customer Observe Customer ID string n/a yes
observe_domain Observe Domain string "observeinc.com" no
observe_token Observe Token string n/a yes
retention_in_days Retention in days of cloudwatch log group number 365 no
s3_bucket Override S3 bucket used to to stage data to be sent to Observe. 
object({
    id  = string
    arn = string
  })
 null no
s3_exported_prefix Key prefix which is subscribed to be sent to Observe Lambda string "" no
s3_lifecycle_rule List of maps containing configuration of object lifecycle management. any [] no
s3_logging Enable S3 access log collection bool false no
snapshot_action List of actions triggered by snapshot. Set to null to inherit all actions supported by the lambda. set(string) 
[
  "autoscaling:Describe*",
  "cloudformation:Describe*",
  "cloudformation:List*",
  "cloudfront:List*",
  "dynamodb:Describe*",
  "dynamodb:List*",
  "ec2:Describe*",
  "ecs:Describe*",
  "ecs:List*",
  "eks:Describe*",
  "eks:List*",
  "elasticache:Describe*",
  "elasticloadbalancing:Describe*",
  "events:List*",
  "firehose:Describe*",
  "firehose:List*",
  "iam:Get*",
  "iam:List*",
  "kinesis:Describe*",
  "kinesis:List*",
  "lambda:List*",
  "logs:Describe*",
  "rds:Describe*",
  "route53:List*",
  "s3:GetBucket*",
  "s3:List*",
  "sns:Get*",
  "sns:List*",
  "sqs:Get*",
  "sqs:List*"
]
 no
snapshot_exclude List of actions to exclude from being executed on snapshot request. list(string) [] no
snapshot_include List of actions to include in snapshot request. list(string) [] no
snapshot_schedule_expression Rate at which snapshot is triggered. Must be valid EventBridge expression string "rate(1 hour)" no
subscribed_log_group_excludes A list of regex patterns describing CloudWatch log groups to NOT subscribe to.

See https://github.com/observeinc/terraform-aws-cloudwatch-logs-subscription#input_log_group_excludes for more info"		 list(string) [] no
subscribed_log_group_filter_pattern A filter pattern for a CloudWatch Logs subscription filter.

See https://github.com/observeinc/terraform-aws-cloudwatch-logs-subscription#input_filter_pattern or
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html for more info"		 string "" no
subscribed_log_group_matches A list of regex patterns describing CloudWatch log groups to subscribe to.

See https://github.com/observeinc/terraform-aws-cloudwatch-logs-subscription#input_log_group_matches for more info"		 list(string) [] no
subscribed_s3_bucket_arns List of additional S3 bucket ARNs to subscribe lambda to. list(string) [] no
tags A map of tags to add to all resources map(string) {} no

Outputs

Name Description
bucket S3 bucket subscribed to Observe Lambda
observe_kinesis_firehose Observe Kinesis Firehose module
observe_lambda Observe Lambda module

License

Apache 2 Licensed. See LICENSE for full details.