etsy/boundary-layer

Inject Python code that will be used in PythonOperator

peleyal opened this issue · 1 comments

peleyal commented

I would like to figure out a way if I can inject some Python code at the beginning of my DAG.
The idea is that I want to use the python_callable field on a PythonOperator to call my code, however my code is going to be like 5-10 lines of code and I cannot really represent it with lambda (or I would like not to, cause it's going to be ugly!)

Currently I see two options but I would like to get your feedback please :) Thanks!

  1. Define a plugin that creates a new operator that inherits from PythonOperator and will do the magic. Then, Boundary Layer will load the YAML file I config in the plugin so the output DAG.py is valid. And, I'll also have to include the implementation itself (not the YAML) in the DAGs folder in airflow itself.
  2. A better option(which I'm not sure if it is supported) will be to have some way to paste code at the beginning of the DAG.py file. This I guess can be done using a pre (or post) processor. However, I'm not sure if Boundary Layer supports something like that.

While exploring these options, I preferred to file a bug and see what you guys think about it.
Thanks!