This repository contains Kafka connector configuration files which enables real time data synchronization from MongoDB to AWS S3 Bucket.
- Kafka Cluster
- MongoDB Instance
- AWS S3 Bucket
-
Download & Install the MongoDB Source & AWS Sink Connector Plugin in your Kafka Cluster
-
Update the following in the
mongodb-source-connector.properties
connector configuration file.CONNECTION-STRING
- MongoDB Cluster Connection StringDB-NAME
- Database NameCOLLECTION-NAME
- Collection Name
-
Update the following in the
s3-sink-connector.properties
connector configuration file.TOPIC-NAME
- Kafka Topic Name (i.e DB.COLLECTION name)S3-REGION
- AWS S3 Region NameS3-BUCKET-NAME
- AWS S3 Bucket Name where you wish to push the data.
-
Deploy the connector configuration files in your Kafka Cluster. This will enable real time data synchronization from MongoDB to AWS S3 Buckets.
Note: The above connector push the data to the S3 bucket at a regular interval of time, these configuration can be modified based on the use case. Refer to the following documentation for more details.