Question: Can i use airflow operators with CWL-Airflow?

Question

Question: Can i use airflow operators with CWL-Airflow?

Amitg1 opened this issue 4 years ago · 1 comments

Can I use airflow operators with CWL-Airflow? or only cmdtools like in CWL?

if so, how I am expected to push new operators or cmdtools to Airflow?

Thanks! great job.

Answer 1 · 2021-07-08T14:24:03.000Z

Hello,
CWL format is more suitable for running command-line tools. And if you are using Docker containers each step will be completely isolated from others, so it's not an easy task to run standard Airflow operator from inside CWL step. I would recommend using something like ExternalTaskSensor for that purpose.

Would you mind describing your use case? Which Airflow Operators are you planning to use and after which workflow steps? Perhaps, this is some common scenario that I'm not aware of and that is worth implementing in CWL-Airflow natively.