/terraform-aws-dms

Terraform module which creates AWS DMS (Database Migration Service) resources 🇺🇦

Primary LanguageHCLApache License 2.0Apache-2.0

AWS DMS Terraform module

Terraform module which creates AWS DMS (Database Migration Service) resources.

Usage

See examples directory for working examples to reference:

module "database_migration_service" {
  source  = "terraform-aws-modules/dms/aws"
  version = "~> 1.0"

  # Subnet group
  repl_subnet_group_name        = "example"
  repl_subnet_group_description = "DMS Subnet group"
  repl_subnet_group_subnet_ids  = ["subnet-1fe3d837", "subnet-129d66ab", "subnet-1211eef5"]

  # Instance
  repl_instance_allocated_storage            = 64
  repl_instance_auto_minor_version_upgrade   = true
  repl_instance_allow_major_version_upgrade  = true
  repl_instance_apply_immediately            = true
  repl_instance_engine_version               = "3.4.5"
  repl_instance_multi_az                     = true
  repl_instance_preferred_maintenance_window = "sun:10:30-sun:14:30"
  repl_instance_publicly_accessible          = false
  repl_instance_class                        = "dms.t3.large"
  repl_instance_id                           = "example"
  repl_instance_vpc_security_group_ids       = ["sg-12345678"]

  endpoints = {
    source = {
      database_name               = "example"
      endpoint_id                 = "example-source"
      endpoint_type               = "source"
      engine_name                 = "aurora-postgresql"
      extra_connection_attributes = "heartbeatFrequency=1;"
      username                    = "postgresqlUser"
      password                    = "youShouldPickABetterPassword123!"
      port                        = 5432
      server_name                 = "dms-ex-src.cluster-abcdefghijkl.us-east-1.rds.amazonaws.com"
      ssl_mode                    = "none"
      tags                        = { EndpointType = "source" }
    }

    destination = {
      database_name = "example"
      endpoint_id   = "example-destination"
      endpoint_type = "target"
      engine_name   = "aurora"
      username      = "mysqlUser"
      password      = "passwordsDoNotNeedToMatch789?"
      port          = 3306
      server_name   = "dms-ex-dest.cluster-abcdefghijkl.us-east-1.rds.amazonaws.com"
      ssl_mode      = "none"
      tags          = { EndpointType = "destination" }
    }
  }

  replication_tasks = {
    cdc_ex = {
      replication_task_id       = "example-cdc"
      migration_type            = "cdc"
      replication_task_settings = file("task_settings.json")
      table_mappings            = file("table_mappings.json")
      source_endpoint_key       = "source"
      target_endpoint_key       = "destination"
      tags                      = { Task = "PostgreSQL-to-MySQL" }
    }
  }

  event_subscriptions = {
    instance = {
      name                             = "instance-events"
      enabled                          = true
      instance_event_subscription_keys = ["example"]
      source_type                      = "replication-instance"
      sns_topic_arn                    = "arn:aws:sns:us-east-1:012345678910:example-topic"
      event_categories                 = [
        "failure",
        "creation",
        "deletion",
        "maintenance",
        "failover",
        "low storage",
        "configuration change"
      ]
    }
    task = {
      name                         = "task-events"
      enabled                      = true
      task_event_subscription_keys = ["cdc_ex"]
      source_type                  = "replication-task"
      sns_topic_arn                = "arn:aws:sns:us-east-1:012345678910:example-topic"
      event_categories             = [
        "failure",
        "state change",
        "creation",
        "deletion",
        "configuration change"
      ]
    }
  }

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

Combinations

Within DMS you can have multiple combinations of resources depending on your use case. For example (not an exhaustive list of possible combinations):

Simple

  • One source endpoint
  • One target/destination endpoint
  • One replication task
  • Two event subscriptions
    • Replication instance event subscription
    • Replication task event subscriptions

DMS Simple

Multiple endpoints, multiple tasks

  • Two source endpoints
  • Three target/destination endpoints
  • Three replication tasks (source1 -> target1, source2 -> target2, source1 -> targe3)
  • Four event subscriptions
    • Replication instance event subscription
    • Replication task event subscription for each task listed above

DMS Complex

In order to accommodate a flexible, multi-resource combinatorial module, keys and maps are used for cross-referencing resources created within the module.

Given the following example (not complete, only showing the relevant attributes):

module "database_migration_service" {
  source  = "terraform-aws-modules/dms/aws"
  version = "~> 1.0"

  # This value is used when subscribing to instance event notifications
  repl_instance_id = "readme-example"

  endpoints = {
    # These keys are used to map endpoints within task definitions by this key `source1`
    source1 = {
      endpoint_type = "source"
      ...
    }

    destination1 = {
      endpoint_type = "target"
      ...
    }

    destination2 = {
      endpoint_type = "target"
      ...
    }
  }

To create replication tasks, you simply reference the relevant keys from the endpoints map in the source_endpoint_key/target_endpoint_key fields:

  ...

  replication_tasks = {
    src1_dest1 = {
      ...
      source_endpoint_key = "source1"
      target_endpoint_key = "destination1"
    }
    src1_dest2 = {
      ...
      source_endpoint_key = "source1"
      target_endpoint_key = "destination2"
    }
  }

  ...

Continuing the same lookup pattern, to create event subscriptions, you simply reference the replication instance ID in the instance_event_subscription_keys field when subscribing to instance notifications, or the replication_tasks keys in the task_event_subscription_keys to subscribe to the tasks notifications (all or only select keys for select tasks):

  ...

  event_subscriptions = {
    instance = {
      instance_event_subscription_keys = ["readme-example"]
      ...
    }
    task = {
      task_event_subscription_keys = ["src1_dest1", "src1_dest2]
      ...
    }
  }

  ...

Tasks

Tasks are the "jobs" that perform the necessary actions of migrating from source to target, including any transformations and/or mappings of the data in transit. Tasks are largely controlled by task settings that are defined in a JSON document.

Replication Task

{
  "TargetMetadata": {
    "TargetSchema": "",
    "SupportLobs": true,
    "FullLobMode": false,
    "LobChunkSize": 64,
    "LimitedSizeLobMode": true,
    "LobMaxSize": 32,
    "BatchApplyEnabled": true
  },
  "FullLoadSettings": {
    "TargetTablePrepMode": "DO_NOTHING",
    "CreatePkAfterFullLoad": false,
    "StopTaskCachedChangesApplied": false,
    "StopTaskCachedChangesNotApplied": false,
    "MaxFullLoadSubTasks": 8,
    "TransactionConsistencyTimeout": 600,
    "CommitRate": 10000
  },
  "Logging": {
    "EnableLogging": false
  },
  "ControlTablesSettings": {
    "ControlSchema":"",
    "HistoryTimeslotInMinutes":5,
    "HistoryTableEnabled": false,
    "SuspendedTablesTableEnabled": false,
    "StatusTableEnabled": false
  },
  "StreamBufferSettings": {
    "StreamBufferCount": 3,
    "StreamBufferSizeInMB": 8
  },
  "ChangeProcessingTuning": {
    "BatchApplyPreserveTransaction": true,
    "BatchApplyTimeoutMin": 1,
    "BatchApplyTimeoutMax": 30,
    "BatchApplyMemoryLimit": 500,
    "BatchSplitSize": 0,
    "MinTransactionSize": 1000,
    "CommitTimeout": 1,
    "MemoryLimitTotal": 1024,
    "MemoryKeepTime": 60,
    "StatementCacheSize": 50
  },
  "ChangeProcessingDdlHandlingPolicy": {
    "HandleSourceTableDropped": true,
    "HandleSourceTableTruncated": true,
    "HandleSourceTableAltered": true
  },
  "ErrorBehavior": {
    "DataErrorPolicy": "LOG_ERROR",
    "DataTruncationErrorPolicy":"LOG_ERROR",
    "DataErrorEscalationPolicy":"SUSPEND_TABLE",
    "DataErrorEscalationCount": 50,
    "TableErrorPolicy":"SUSPEND_TABLE",
    "TableErrorEscalationPolicy":"STOP_TASK",
    "TableErrorEscalationCount": 50,
    "RecoverableErrorCount": 0,
    "RecoverableErrorInterval": 5,
    "RecoverableErrorThrottling": true,
    "RecoverableErrorThrottlingMax": 1800,
    "ApplyErrorDeletePolicy":"IGNORE_RECORD",
    "ApplyErrorInsertPolicy":"LOG_ERROR",
    "ApplyErrorUpdatePolicy":"LOG_ERROR",
    "ApplyErrorEscalationPolicy":"LOG_ERROR",
    "ApplyErrorEscalationCount": 0,
    "FullLoadIgnoreConflicts": true
  }
}

Examples

Examples codified under the examples are intended to give users references for how to use the module(s) as well as testing/validating changes to the source code of the module. If contributing to the project, please be sure to make any appropriate updates to the relevant examples to allow maintainers to test your changes and to keep the examples up to date for users. Thank you!

Requirements

Name Version
terraform >= 0.13.1
aws >= 4.17
time >=0.7.2

Providers

Name Version
aws >= 4.17
time >=0.7.2

Modules

No modules.

Resources

Name Type
aws_dms_certificate.this resource
aws_dms_endpoint.this resource
aws_dms_event_subscription.this resource
aws_dms_replication_instance.this resource
aws_dms_replication_subnet_group.this resource
aws_dms_replication_task.this resource
aws_iam_role.dms_access_for_endpoint resource
aws_iam_role.dms_cloudwatch_logs_role resource
aws_iam_role.dms_vpc_role resource
time_sleep.wait_for_dependency_resources resource
aws_iam_policy_document.dms_assume_role data source
aws_iam_policy_document.dms_assume_role_redshift data source
aws_partition.current data source

Inputs

Name Description Type Default Required
certificates Map of objects that define the certificates to be created map(any) {} no
create Determines whether resources will be created bool true no
create_iam_roles Determines whether the required DMS IAM resources will be created bool true no
create_repl_subnet_group Determines whether the replication subnet group will be created bool true no
enable_redshift_target_permissions Determines whether redshift.amazonaws.com is permitted access to assume the dms-access-for-endpoint role bool false no
endpoints Map of objects that define the endpoints to be created any {} no
event_subscription_timeouts A map of timeouts for event subscription create/update/delete operations map(string) {} no
event_subscriptions Map of objects that define the event subscriptions to be created any {} no
iam_role_permissions_boundary ARN of the policy that is used to set the permissions boundary for the role string null no
iam_role_tags A map of additional tags to apply to the DMS IAM roles map(string) {} no
repl_instance_allocated_storage The amount of storage (in gigabytes) to be initially allocated for the replication instance. Min: 5, Max: 6144, Default: 50 number null no
repl_instance_allow_major_version_upgrade Indicates that major version upgrades are allowed bool null no
repl_instance_apply_immediately Indicates whether the changes should be applied immediately or during the next maintenance window bool null no
repl_instance_auto_minor_version_upgrade Indicates that minor engine upgrades will be applied automatically to the replication instance during the maintenance window bool null no
repl_instance_availability_zone The EC2 Availability Zone that the replication instance will be created in string null no
repl_instance_class The compute and memory capacity of the replication instance as specified by the replication instance class string null no
repl_instance_engine_version The engine version number of the replication instance string null no
repl_instance_id The replication instance identifier. This parameter is stored as a lowercase string string null no
repl_instance_kms_key_arn The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt the connection parameters string null no
repl_instance_multi_az Specifies if the replication instance is a multi-az deployment. You cannot set the availability_zone parameter if the multi_az parameter is set to true bool null no
repl_instance_preferred_maintenance_window The weekly time range during which system maintenance can occur, in Universal Coordinated Time (UTC) string null no
repl_instance_publicly_accessible Specifies the accessibility options for the replication instance bool null no
repl_instance_subnet_group_id An existing subnet group to associate with the replication instance string null no
repl_instance_tags A map of additional tags to apply to the replication instance map(string) {} no
repl_instance_timeouts A map of timeouts for replication instance create/update/delete operations map(string) {} no
repl_instance_vpc_security_group_ids A list of VPC security group IDs to be used with the replication instance list(string) null no
repl_subnet_group_description The description for the subnet group string null no
repl_subnet_group_name The name for the replication subnet group. Stored as a lowercase string, must contain no more than 255 alphanumeric characters, periods, spaces, underscores, or hyphens string null no
repl_subnet_group_subnet_ids A list of the EC2 subnet IDs for the subnet group list(string) [] no
repl_subnet_group_tags A map of additional tags to apply to the replication subnet group map(string) {} no
replication_tasks Map of objects that define the replication tasks to be created any {} no
tags A map of tags to use on all resources map(string) {} no

Outputs

Name Description
certificates A map of maps containing the certificates created and their full output of attributes and values
dms_access_for_endpoint_iam_role_arn Amazon Resource Name (ARN) specifying the role
dms_access_for_endpoint_iam_role_id Name of the IAM role
dms_access_for_endpoint_iam_role_unique_id Stable and unique string identifying the role
dms_cloudwatch_logs_iam_role_arn Amazon Resource Name (ARN) specifying the role
dms_cloudwatch_logs_iam_role_id Name of the IAM role
dms_cloudwatch_logs_iam_role_unique_id Stable and unique string identifying the role
dms_vpc_iam_role_arn Amazon Resource Name (ARN) specifying the role
dms_vpc_iam_role_id Name of the IAM role
dms_vpc_iam_role_unique_id Stable and unique string identifying the role
endpoints A map of maps containing the endpoints created and their full output of attributes and values
event_subscriptions A map of maps containing the event subscriptions created and their full output of attributes and values
replication_instance_arn The Amazon Resource Name (ARN) of the replication instance
replication_instance_private_ips A list of the private IP addresses of the replication instance
replication_instance_public_ips A list of the public IP addresses of the replication instance
replication_instance_tags_all A map of tags assigned to the resource, including those inherited from the provider default_tags configuration block
replication_subnet_group_id The ID of the subnet group
replication_tasks A map of maps containing the replication tasks created and their full output of attributes and values

License

Apache-2.0 Licensed. See LICENSE.