Add support for reusable templates -> workflows

[samsmithnz]

Add support for reusable templates -> workflows

[calebcartwright]

Hey Sam 👋 is this issue intended to (eventually) address this https://github.com/samsmithnz/AzurePipelinesToGitHubActionsConverter#templates or is "templates" referring to something different here?

[samsmithnz]

@calebcartwright hey there! 👋 Yes this is the right issue, but it's not straight forward, as they don't translate over directly and are handled differently in GitHub Actions. Still not sure on the best translation. Any thoughts?

[calebcartwright]

Yeah I originally came over here with the assumption it wasn't supported as it seems like a really hard thing to even attempt.

My intuition at this point would be that the only remotely feasible option for semi-converting templates would be to attempt to resolve template references and then run your own expansion engine over them to get the resultant stage/job/steps that could then be converted. However that also sounds like a royal pain!

Would be super helpful if some of your former colleagues would consider open sourcing or sharing the internals of how Azure DevOps does that today to generate the build plan 😁

[samsmithnz]

Yes - makes sense and agree. To be clear - this will convert templates very well today- but it's the reusable workflow part that is more challenging.

[calebcartwright]

Gotcha, thanks for the clarification. Seems like a massive undertaking but on the off chance I ever find myself with a couple months of consecutive free time...

[ChristopherHX]

Would be super helpful if some of your former colleagues would consider open sourcing or sharing the internals of how Azure DevOps does that today to generate the build plan 😁

GitHub did share some internals of Azure Pipelines in https://github.com/actions/runner ( 2019 ), the first commit contains more code from Azure Pipelines than later versions. Published some months after GitHub forked the codebase.

This includes a deriviate of the Azure Pipelines Template Engine

• Without each, if, elseif and else directives.
• Uses expressions v2 from Github Actions instead of expressions v1 from Azure Pipelines, however the expressions v1 source code was also published

this will convert templates very well today

Doesn't this convert them to empty jobs and skips the templates references? I don't think it is possible convert fully to reusable workflows, because they don't support step and job injection. Also reusable workflows jobs are scoped and the parent workflow have no direct access to the jobs defined in the reusable workflow.

Based on actions/runner I created an experimental Azure Pipelines Template Expander (https://github.com/ChristopherHX/runner.server/blob/302aed4e936d06e8dbd58c0082dcaa902f97ef88/src/Runner.Server/AzureDevops.cs#L401 / https://github.com/ChristopherHX/runner.server/blob/302aed4e936d06e8dbd58c0082dcaa902f97ef88/src/Runner.Server/Pipeline.cs#L17), it resolves template references including remote templates and merges the Pipeline into a single instance.

Instead of converting Azure Pipelines to GitHub Actions I emulate a whole Azure Pipelines Server and use the official Agents to execute the job steps locally.

I guess it could be helpful to use my code to preprocess the pipelines into a flat yaml file without template references and template expressions.
One problem is the different .net Target .netstandard2.0 vs .net6.0.
It is probably a challenge to make this component reuable as a single nuget package without reorganize the project structure of my actions/runner fork.

My engine is able to evaluate

steps:
  - ${{ each x in split('hello, world, x, y', ', ') }}:
      - script: |
          echo ${{ x }}
      - ${{ if in(x, 'x', 'y') }}:
          - script: |
              echo "This step is injected if x is 'x' or 'y'"
      - ${{ else }}:
          - script: |
              echo "This step is injected if x isn't 'x' or 'y'"

to

steps:
  - script: |
      echo hello
  - script: |
      echo "This step is injected if x isn't 'x' or 'y'"
  - script: |
      echo world
  - script: |
      echo "This step is injected if x isn't 'x' or 'y'"
  - script: |
      echo x
  - script: |
      echo "This step is injected if x is 'x' or 'y'"
  - script: |
      echo y
  - script: |
      echo "This step is injected if x is 'x' or 'y'"