YotpoLtd/metorikku

Control writing specific output in yaml file

akizminet opened this issue · 1 comments

I have some jobs that need to output multiple dataframes. However, I also want to control output by using environment variables. Is there any option available for controlling writing dataframes?

Rigth now you could force using environment varaibles such as boolean to create empty dataframes. For example, you could query your dataframe as

SELECT *
FROM <your_data_frame>
WHERE ${create_dataframe} = 'true'

And in job_config.yaml you could create this var as

variables:
  create_dataframe: true

Finally, on output options set protectFromEmptyOutput to true