applications: Create AWS Glue ETL Boilerplate
Closed this issue · 1 comments
ulises-jeremias commented
Describe the feature
Use Case
Integrate several data sources using AWS Glue
Proposed Solution
.
docs/
local-dev/
resources/ # cloud formation resources
libs/
jobs/
etl/
foo_src_bar_dest_job.py
. . .
docker-compose.yml
serverless.yml
glue.yml
Other Information
No response
Acknowledgements
- I may be able to implement this feature request
- This feature might incur a breaking change
Version used
Environment details (OS name and version, etc.)
edwinabot commented
So far, I've been able to put together a local development environment to test glue jobs end-to-end.
Most relevant capabilities so far:
- Test reads from S3 (mocking it locally using Localstack)
- Test writes to PostgreSQL
- Test writes to DocumentDB (mocking it locally using MongoDB 4.0)