Copying files from AWS S3 to Azure Blob (ADLS Gen2) using ADF.
Create the infrastructure:
terraform -chdir="infra" init
terraform -chdir="infra" apply -auto-approve
Generate the dummy data:
bash generateDummyData.sh
Upload the files to AWS S3:
# Change the bucket name
aws s3 sync ./dummydata s3://YOUR_BUCKET/data
# List objects
aws s3 ls s3://YOUR_BUCKET --recursive --human-readable --summarize
To finish the process, log in to ADF and create a copy pipeline.