[ENG-2816] [FEATURE REQUEST] Automatically create directory when creating S3 Integration
Closed this issue · 7 comments
Problem
I'm attempting to create a new AWS S3 configuration that will store the aqueduct metadata. I'd like to use a subdirectory for the storage, but I get the following error when connecting the integration:
Exception: Supplied root directory aqueduct/ does not exist in bucket <bucket name>.
Solution
It'd be beneficial to have the subdirectory automatically created if it does not exist.
Alternatives
If having the subdirectory automatically created is not desirable, having it as a optional checkbox underneath directory would also be a possible solution.
Giving this to Jerome to get him exposure to the python executor operators.
@jerome take a look around line 42 of aqueduct_executor/operators/connectors/data/s3.py
Ignore: test comment to verify Linear & github syncing
@jpurusho65 seeking confirmation that this PR Is still open and did not make the Mon 11PM cut-off.
Kindly update stale due date and release label accordingly.
Please also share what kind of complexity is holding up this PR - out for review from 8 days ago.
I have moved the due data to tomorrow. We had some back & forth on test integration for this, and found it was not straightforward to add that. Decided to move that change to a separate issue.
The primary change itself is good and was verified with our UI early on, and confident to push yesterday, but got held up because of adding s3 integration test.
Created new issue ENG-2946 to track test changes