nanlabs/aws-glue-etl-boilerplate

applications: Create AWS Glue ETL Boilerplate

Closed this issue · 1 comments

Describe the feature

Use Case

Integrate several data sources using AWS Glue

Proposed Solution

.
   docs/
   local-dev/
   resources/     # cloud formation resources
   libs/
   jobs/
       etl/
       foo_src_bar_dest_job.py
       . . .
   docker-compose.yml
   serverless.yml
   glue.yml

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

Version used

Environment details (OS name and version, etc.)

So far, I've been able to put together a local development environment to test glue jobs end-to-end.

Most relevant capabilities so far:

  • Test reads from S3 (mocking it locally using Localstack)
  • Test writes to PostgreSQL
  • Test writes to DocumentDB (mocking it locally using MongoDB 4.0)