- Create a S3 Bucket
- Create a ECR repository
add a .env
with the following credentials
BUCKET=
ECR=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
AWS_ACCOUNT_ID=
- Create a ECS cluster
- Create a task for each spider and schedule it.
add a .env
with the following credentials
BUCKET=
ECR=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
AWS_ACCOUNT_ID=