Refactor pipelines to be more modular and still easy to review
tvogels01 opened this issue · 0 comments
This is triggered by PR #15 . The rebuild, refresh and pizza loader pipelines are very similar.
(1) Can we merge the rebuild and refresh pipeline definitions? This would add the feature to the rebuild of being able to specify some specific schema to rebuild. This appears desirable for development work. Now that we have arthur in the mix for evaluating variables, the merge might be possible without much loss of readability.
(2) Should we add a feature to the rendering code in Arthur that allows (at least one level) of expansion where one template may refer to another. This way, we could break up the pipeline definitions into smaller fragments and then pull in to the actual pipeline definition the fragments that are needed.