Allow Customizing File Write Behavior of export_to_file Operator
Opened this issue · 0 comments
jordanyakerirt commented
Please describe the feature you'd like to see
The current implementation of the export_to_file operator doesn't provide any mechanism for overriding or customizing the behavior of the file output. Instead the operator simply calls one of a number of Pandas.DataFrame.to_*
functions with default functions that SDK users are locked in to.
Describe the solution you'd like
Allow the passing in of a dictionary of parameters that get conveyed to the underlying Pandas.DataFrame
functionality.
Are there any alternatives to this feature?
No. The current write implementation is hard-coded to a specific set of parameters in the FileType.create_from_dataframe
functionality.
Acceptance Criteria
- All checks and tests in the CI should pass
- Unit tests (90% code coverage or more, once available)
- Integration tests (if the feature relates to a new database or external service)
- Example DAG
- Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
- Exception handling in case of errors
- Logging (are we exposing useful information to the user? e.g. source and destination)
- Improve the documentation (README, Sphinx, and any other relevant)
- How to use Guide for the feature (example)