This project aims to deploy a Change Data Capture (CDC) solution using Terraform. The CDC solution utilizes AWS services such as AWS Database Migration Service (DMS), Amazon S3, AWS Lambda, and AWS Glue.
Before getting started, make sure you have the following prerequisites installed:
- Terraform: Installation Guide
- AWS CLI: Installation Guide
-
Clone the repository:
git clone https://github.com/your-username/project-big-data.git
-
Navigate to the project directory:
cd project-big-data/cdc_project
-
Initialize Terraform:
terraform init
-
Configure your AWS credentials:
aws configure
-
Modify the
variables.tf
file to customize your deployment settings. -
Deploy the infrastructure:
terraform apply
-
Monitor the deployment progress and wait for it to complete. This process may take up to 10 minutes.
The CDC project architecture consists of the following components:
-
AWS Database Migration Service (DMS): Used to migrate data from the source database to Amazon S3.
-
Amazon S3: Serves as the storage for the migrated data and the output files generated by AWS Glue.
-
AWS Lambda: Triggers AWS Glue when DMS migrates data to Amazon S3.
-
AWS Glue: Handles the data transformation and processing jobs, and saves the output files back to Amazon S3.
- AWS DMS and Amazon S3:
- Check AWS DMS UI and S3 UI to verify that the data has been successfully migrated to S3.
- AWS Lambda:
- Lambda function executions and any print statements will be logged in AWS CloudWatch. Go to the Lambda UI and select monitor to access AWS CloudWatch logs.
- AWS Glue:
- After lambda function execution, navigate to the Glue UI to see running jobs. Glue job logs can be found in AWS CloudWatch.
- Data Update Cycle:
- Whenever there are insertions, updates, or deletions, DMS will migrate that data to S3, initiating the cycle again.
When you're done, you can destroy the resources created by Terraform.
terraform destroy