astronomer/astro-sdk

Allow Customizing File Write Behavior of export_to_file Operator

Opened this issue · 0 comments

Please describe the feature you'd like to see
The current implementation of the export_to_file operator doesn't provide any mechanism for overriding or customizing the behavior of the file output. Instead the operator simply calls one of a number of Pandas.DataFrame.to_* functions with default functions that SDK users are locked in to.

Describe the solution you'd like
Allow the passing in of a dictionary of parameters that get conveyed to the underlying Pandas.DataFrame functionality.

Are there any alternatives to this feature?
No. The current write implementation is hard-coded to a specific set of parameters in the FileType.create_from_dataframe functionality.

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)