actions/runner

Next Steps for Fully Functioning Composite Actions

ethanchewy opened this issue ยท 102 comments

NOTE:

For those who are not aware, I'm a college student who interned at GitHub for Summer 2020. Thank you all for all the thoughtful comments and feedback on this thread but since I'm not working at GitHub currently, I will not be responding to any new comments below in this thread.

================================================

Hi there ๐Ÿ‘‹

We just released a new feature called composite run steps actions(https://docs.github.com/en/actions/creating-actions/about-actions#composite-run-steps-actions).

This issue thread will serve as a point of reference for people who are curious about the next steps after this feature and what we plan to work on next. There are probably quite a few questions about when we are going to support in the future like "when are you going to support 'uses' steps in a composite action"? In this Issue thread, I'll talk about the technical problems that we are planning to solve to accomplish this. Throughout the next week, I'll update it with any additional updates.

tl;dr in this issue, I'll go over what we currently support for composite run steps and how we plan on developing fully functioning composite actions.

Composite run steps background

What is a composite run steps action?

Many of you have been asking us to build a feature that enables them to nest actions within actions (ex: #438).

An important first step for supporting nesting within action is to first start supporting the execution of multiple run steps in an action.

For some developers, they may want to execute a variety of scripts and commands in different languages and shells in a single action and map those outputs to the workflow. Weโ€™ve also heard from our community that many developers want to reuse parts of their workflows into other workflows. This new action type, composite run steps, can help developers do just that! For example, developers can abstract parts of their workflows as composite run steps actions and reuse them for other workflows.

Thus, we created this feature called composite run steps which allows users to run multiple run steps in an action!

What does composite run steps currently support?

For each run step in a composite action, we support:

  • name
  • id
  • run
  • env
  • shell
  • working-directory

In addition, we support mapping input and outputs throughout the action.

See docs for more info.

What does Composite Run Steps Not Support

We don't support setting conditionals, continue-on-error, timeout-minutes, "uses", and secrets on individual steps within a composite action right now.

(Note: we do support these attributes being set in workflows for a step that uses a composite run steps action)

Examples of how you can use composite run steps actions

Next Steps for Developing Fully Functioning Composite Run Steps

In this section, I'll describe some current features that we are currently working on or are planning to work on. You can see more details about each of these in https://github.com/actions/runner/blob/users/ethanchewy/compositeADR/docs/adrs/0549-composite-run-steps.md.

If conditionals

see: https://github.com/actions/runner/blob/main/docs/adrs/0549-composite-run-steps.md#if-condition

For if conditionals, we treat the composite action as a completely new slate. Each proceeding step depends on the proceeding step's states (failure, cancelled, or success).

To keep track of these states, we can create a new ExecutionContext for composite actions that keep track of the attributes we need such as the current state of a composite action step before evaluating. When we support nesting, we need to follow some sort of inheritance model (maybe something like state = (parent state && child state)?

Uses

Related questions that we need to answer first before we can support uses:

  • How are we going to handle post and pre steps? Should we aggregate them into one individual step for each type?
  • When should we download the repos for nested actions

Current experimentation for this: #612

timeout-minutes

It makes a ton of sense creating a separate execution context for composite actions especially in the cases for keeping track of the conditional status of the step. This would make it easier to some sort of "time element" to easily get the "correct" condition of the step in log time (something like this algorithm: https://maksimdan.gitbook.io/interview-practice-problems/leetcode_sessions/design/permission-management-for-a-general-tree)

For the nested timeout-minutes support, couldn't we just link all children tokens to their parents? The time left will trickle down to the children. For example,

|A (30 min.)|
  [B (15 min.)|
    |B2|
  |C (None)|
    |C2|

Let's say the A composite action whole step has a timeout-minutes fo 30 minutes and the step B (which is a step inside the composite action) has a timeout-minutes of 15 minutes and the step B exceeds that time, we can easily fail that thread and differentiate that between a cancellation. Then, we would move onto the next step C which doesn't have a timeout-minutes set and since it has ~15 minutes left, it will still start running. If the next step C takes more than 15 minutes, then we fail the whole action since the whole action has taken more than 30 minutes (the step C will automatically fail as well as the whole composite action step since they are linked)

Areas that we are also thinking about

There are many more layers of complexity for fully functioning composite actions.

For example at a high level,

  • How would we use multiple Docker actions in a composite action effectively?
  • How do we propagate and resolve errors effectively with nested actions?
  • Should we support the shell attribute for a uses step? (probably not)

Runner specific questions:

  • Should we create a separate, slimmed-down version of ExecutionContext for Composite Actions?
    • Yes, because this would reduce confusion and create an easier way to add onto it + easily hook this up with the .Global ExecutionContext.
  • How do we map our Scopes in nested actions?
    • We've already come up with a pretty clever solution for generating a context name (i.e. {context.ScopeName}.{context.ContextName})
    • Our dictionary for resolving outputs is in the nested dictionary StepsContext[Step_Name][Scope_Name]["outputs"]. Could we flatten the dictionary to reduce lookup time?

TODOs

  • Handle Continue-on-error (we need to change how we handle failures in the composite action handler [i.e. don't call Complete(TaskResult.Failed))
  • Handle private actions
  • Outcomes?
  • Revisit defaults? Does it make sense to add defaults once we have fully functional composite actions.

Extra Information

How we built it

will reopen when feature launches.

Thanks for the release! Can composited actions be referenced from the local repository (in the way standard actions can https://docs.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#referencing-an-action-in-the-same-repository-where-a-workflow-file-uses-the-action) or can they only be referenced from separate public repositories?

Great question @latacora-tomekr . Yes, this can be referenced from the local repository as well.

Thank you @latacora-tomekr for that pointer. I was trying to follow this example:

https://docs.github.com/en/actions/creating-actions/creating-a-composite-run-steps-action

And it wasn't seeing the action. I tried it with the relative path, and it worked in my dummy repo. I will leave this up, despite my embarrassment, in the hopes it will help others:

https://github.com/jeffeb3/hello-world-composite-run-steps-action

Thank you for this feature! I was wondering today if it would be possible to reuse Github Actions yaml steps and found out this was just released. Talk about great timing!

Aside from what's already in the roadmap (uses, visualization), it would be great to have action scope env and shell keywords to allow specifying action scope step shell and action scope environment variables.

Thanks again for working on this!

This is very useful and fits my use case well: I mostly run shell scripts in my workflows, and making them reusable is now a lot simpler.

I've noticed that the "Scripts" example uses an environment variable $GITHUB_ACTION_PATH to access files living in the same directory tree as the action itself. This isn't documented anywhere, neither under Creating a composite steps action nor under Default environment variables, and should probably be amended?

This is very useful and fits my use case well: I mostly run shell scripts in my workflows, and making them reusable is now a lot simpler.

I've noticed that the "Scripts" example uses an environment variable $GITHUB_ACTION_PATH to access files living in the same directory tree as the action itself. This isn't documented anywhere, neither under Creating a composite steps action nor under Default environment variables, and should probably be amended?

@bewuethr Thanks for the kind words!

github.action_path is documented here: https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context (see "github.action_path). github.action_path is equivalent to GITHUB_ACTION_PATH.

We follow this same pattern of also adding the github context attribute as an environment variable like github.token = GITHUB_TOKEN.

Hi, this is really great. Maybe I missed it, am I right in saying the composite action must be in a public repo right now? If so, is there a plan for referencing private actions? We don't really want to make our all CICD scripts public just to reuse them and we have many repos that all need to reference central pipelines / scripts / actions.

@mikeparker good point - I'll add it to our tracking issue so that we can think it through. Thanks for bringing this up!

@mikeparker @ethanchewy I can see that issue on the roadmap for Q1 2021.

Steps in workflow are shown as different collapsible sections in the UI. Are there any plans for supporting similar UI (nested within the workflow's uses step) for composite actions? Something like core.group() for each composite step would be nice for starters ๐Ÿ™
Right now, I'm manually setting the group workflow command in each step.

It seems that we cannot use the outcome of a composite step in an action, is that expected?
outputs: output_var: description: 'Outcome of running pretty format on the code' value: ${{ steps.step_id.outcome }}

and
- run: echo "this is ${{ steps.step_id.outcome }} ${{ steps.step_id_2.outcome }} ${{ steps.step_id_3.outcome }}" shell: bash

both of the above code snippets have empty values for the ${{ }} parameters.

The output of toJson(steps) was an empty Json array {}

Also, it seems that composite actions do not support defaults. https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#defaultsrun

Is this something planned? I notice its not in the initial description of this issue.

It seems that we cannot use the outcome of a composite step in an action, is that expected?
outputs: output_var: description: 'Outcome of running pretty format on the code' value: ${{ steps.step_id.outcome }}

and
- run: echo "this is ${{ steps.step_id.outcome }} ${{ steps.step_id_2.outcome }} ${{ steps.step_id_3.outcome }}" shell: bash

both of the above code snippets have empty values for the ${{ }} parameters.

The output of toJson(steps) was an empty Json array {}

Outcomes are not supported in composite steps at the moment. I'll update my description above - we should support them once we add conditionals.

Also, it seems that composite actions do not support defaults. https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#defaultsrun

Is this something planned? I notice its not in the initial description of this issue.

Defaults are not supported in composite run steps actions.

The reason why we avoided defaults for the design is that we want the workflow author to not know any of the internal workings of the composite action yaml file. So, we require the composite action author to always explicitly define shell for each run step in a composite action.

Can you add support for third party actions in the composite action.
today composite action only supports a group of shell steps, it's less useful for most of our cases.

So my use case is that I have a monorepo.

For each package inside the monorepo I want to execute a composite action (which does build, cache, artifacts and deployment...). I don't want to repeat myself over and over again, that's why. I would use a workflow template for that, BUT they all have shared dependencies in common that have to built first.

What features are needed?

  • ability to create composite of actions
  • ability to trigger composite only on certain file changes

Can we expect those features in the near future?

This is great, really appreciate the feature. The one area I would like to see expanded is the ability to use Javascript step in an action. I use actions/github-script@v3 with nearly identical code in 50 private repos and having a composite would simplify that immensely. I know I should just create an action, but sometimes we have minor difference that multiple teams share.

this is a great feature.

does each composite run action need their separate repository or is it possible to have a single repository with all composite actions? It is not clear to me from the documentation if this is possible and how it can be used

I have an issue with self-hosted runners with custom docker containers. They do not find the composite actions.
Reason: github.action_path is incorrect.
It should be prefixed /__w/_actions but instead it is prefixed /home/github-runner/actions-runner/_work/_actions
See here: #716

Can composite actions populate ACTIONS_RUNTIME_URL and ACTIONS_RUNTIME_TOKEN env variables?

Thanks everyone for the thoughtful comments! :)

Just for full transparency, I interned at GitHub last summer and no longer work there since I'm currently finishing my last year at college as a full-time student.

I'll leave this open but I won't be able to respond to any comments since I am not currently working at GitHub!

@j-carvalho AFAICT, a single repository can contain multiple composite actions, BUT only the subtrees of those actions are fetched.

I wanted to follow a pattern like github/codeql-action/analyze@v1, github/codeql-action/autobuild@v1. That works, but the runner only downloads the ./analyze/** path - without the full repository the composite actions can't share code.

I think fetching the entire repository would be a nice feature.

Is there any timeline for when uses in a composite action will be supported? Without this, it's not possible to invoke, for example, the actions/cache action, so it's not possible to create composite actions that make use of the cache.

I โค๏ธ using composite actions!

For me, the primary reason for using composite action is the reuse of these actions across multiple jobs and workflows.

However, there is one thing that I am concerned about, and it would be nice to see if anyone else has similar concerns. When we use a composite action with multiple steps in a workflow, it appears in the output of the resulting workflow as if these three steps are inlined into a single workflow step.

Instead, I would prefer to see these three steps in the resulting workflow's output as if we had defined these steps in the workflow itself.

How do you feel about that?

@localheinz That's a concern for me too (though, not as big as invoking actions within composite actions).

I think the display could be nested: at the top level, the action is shown as a single step that you can expand to see all the internal steps of this action. Then you can expand all these internal steps to see their logs.

This could extend over multiple levels if there are "actions in actions in actions", etc. (once nested actions are implemented).

Is there any timeline for when uses in a composite action will be supported? Without this, it's not possible to invoke, for example, the actions/cache action, so it's not possible to create composite actions that make use of the cache.

+1 on this. The proof of concept code in #612 was closed almost a month ago, but having even partial or alpha support would be valuable.

Kampe commented

Thanks everyone for the thoughtful comments! :)

Just for full transparency, I interned at GitHub last summer and no longer work there since I'm currently finishing my last year at college as a full-time student.

I'll leave this open but I won't be able to respond to any comments since I am not currently working at GitHub!

Great work! Thank you Ethan!

Has this initiative been picked up by anyone else still currently employed? Given the activity on this issue today alone I'd say some of these requests are highly anticipated!

I see uses mentioned here as unsupported. Does that mean it will be handled outside of this ticket or is there another interpretation?

Can we get a status update on this? Where does this land on the roadmap? Looks like @ethanchewy no longer works for github and the ticket is unassigned.

@mikedrexler, @bryanmacfarlane, @lanecm

I think the design of reusable components of workflows is headed in the right direction with composite actions; however, as it stands currently, composite actions are less attractive in most cases I can think of than simply writing and maintaining a shell script in the client repo. For one, you can debug it (locally in your shell) without having to run it through a GHA trigger. Ultimately, you will have to run the checkout action to be able to get locally-defined composite actions anyway; the same amount of boilerplate would be needed for steps executed directly in a shell script you have plopped into your project, executable after running the checkout action.

For me, this is only a value-add once uses: (actions called via composite actions) is implemented. Most of the boilerplate for workflow files (bootstrapping the container for deployments, notifications, etc.) comes from the GHA-specific accommodations, such as environmental setup (python, node) and organization of auth creds (via secrets). Most of these require uses. For parts of the process such as build, yes, that code could be written in a GHA composite action, but it could just as well (and preferably would be) written in an actual shell script (that then is simply called within a GHA workflow file) instead of being sprinkled throughout a workflow file, and rendered as clumsily coupled to GHA's environment.

Besides those GHA-specific accommodations which are necessary in most workflow files, without uses: the other big value-loss is the ability to make use of 3rd party actions which have complex (but for a project's purpose, static/never-changing), noisy configuration, which you would, as a CI/CD maintainer, prefer to keep quiet underneath an abstraction layer once implemented. It's basically impossible to leverage composite actions in their current state for a purpose for which there isn't currently a better method.

I like the design, I think the initial implementation is a good POC, and I hope uses: is implemented soon. There are a lot of things I'd like to do for my team which are currently unfeasible, due to the maintenance burden that GHA's workflow file patterns suffers from without a uses:.

I think this shows promise in solving the same problems that the talk of re-using setup containers for multiple subsequent jobs within the same workflow file, and YAML:anchors/aliases sought to solve, more effectively, and fulfillment of this functionality would be the better route. At least as far as all my personal use cases can see.

A very specific example. A non-composite container action does not allow customising the Dockerfile/image.

runs:
  using: 'docker'
  image: 'Dockerfile'
  args:
    - ${{ inputs.run_file }}

A composite action allows doing so:

runs:
  using: "composite"
  steps:
    - run: docker run --rm -v $(pwd):/src -w /src ${{ inputs.image }} ${{ inputs.cmd }}
      shell: bash

But it would be desirable to use uses: instead of run:

runs:
  using: "composite"
  steps:
    - uses: docker://${{ inputs.image }}
      with:
        args: ${{ inputs.cmd }}

re-using setup containers for multiple subsequent jobs within the same workflow file, and YAML:anchors/aliases sought to solve, more effectively, and fulfillment of this functionality would be the better route.

I'd prefer to have both things. The YAML anchors thing appears to be just parsing upgrading matter so why not ship it first. It's in my opinion a little unusual that the devops/developers-focused service like GH Actions has not enabled the YAML anchors functionality from day one.

I don't want to quote @dougpagani's comment in full, as it's long, but is there any timeline for supporting uses? This feels like a pretty crucial step in the ecosystem as it opens the possibility of building on top of existing actions.

By way of example, at DoltHub we are implementing Discord notifications via webhooks for all of our infra. This action is "good enough", but it isn't usable without wrapping it around something more succinct to call in our actual actions.

People have followed-up to my comment with examples, and so I thought I'd add a couple to highlight how much of a value-add uses: is.

First example: notifications

Imagine you want to send email notifications. This action would be a cheap way to do it without having to add the code yourself to the codebase: https://github.com/marketplace/actions/send-email

Cruft

- name: Send mail
  uses: dawidd6/action-send-mail@v2
  with:
    server_address: smtp.gmail.com
    server_port: 465
    username: ${{secrets.MAIL_USERNAME}}
    password: ${{secrets.MAIL_PASSWORD}}
    subject: Github Actions job result
    # Literal body:
    body: Build job of ${{github.repository}} completed successfully!
    # Read file contents as body:
    body: file://README.md
    to: obiwan@tatooine.com,yoda@dagobah.com
    from: Luke Skywalker # <user@example.com>
    # Optional carbon copy recipients
    cc: kyloren@starkiller.com,leia@alderaan.com
    # Optional blind carbon copy recipients
    bcc: r2d2@jakku.com,hansolo@milleniumfalcon.com
    # Optional content type (defaults to text/plain):
    content_type: text/html
    # Optional converting Markdown to HTML (set content_type to text/html too):
    convert_markdown: true
    # Optional attachments:
    attachments: attachments.zip,git.diff,./dist/static/main.js

Now, once you've configured it, no matter when you send an email (deployment, build break, job break, w.e.), you'll probably use 9/10 of the same configuration options, just needing to change the email payload. So you will have to copy-paste that into your 20-or-so workflow files anytime you want to send a simple email. And then, if you change the email client or the options or the "default" email-subject, you'll have to update it in those 20-or-so places. If we had uses:, we could cut down on all that noise (and hard-to-maintain-cruft) with:

Clean

- uses: ./.github/actions/send-email # repo-local gha reference
  with: 
    payload: ${{ steps.format-payload.outputs.payload }}

... making it easy for both the maintainer of cicd, and any project-dev that wants to re-use that functionality.

For this example, the notification provider is just an email. But I imagine a "Post to Slack" action would similarly have a lot of one-time configuration cruft you'd want to abstract away into a local action just the same.

Second Example: Bootstrapping a developer or opsperson environment

For our projects, and I imagine others are similar in this respect, we have a good amount of setup we need to do in each and every single workflow to get it to be useful, and that never varies. For us, we have to:

Cruft

  • setup node
  • setup python
  • check gha-cache for npm deps
  • install npm dependencies (if cache miss)
  • grab configuration data (envars) for this commit
    ... (and for builds/deploys)
  • do build
  • save intermediate build artifacts

... just so we can build & test, or build & deploy. I'm not going to write out all the yaml for this example, because it would be 100s of lines, but this too could be easily reduced to a 2-or-3-line preamble that looks like:

Clean

- uses: ./.github/actions/bootstrap-dev-env # repo-local gha reference
  with: 
    cache-key: ${{ cache-key-generated-from-some-hashing-of-relevant-files }} 
    other-special-config: VALUE

@dougpagani's comment definitely pointed out how much value-add uses can create. I want to put my personal thoughts on how it should/could be implemented.

Proposal

composite should be a convention/macro, not an encapsulation. What does it mean?

  • composite is simply a convention/macro to put multiple steps together into a single piece of reusable code.
  • Before a workflow runs, all composite actions are replaced with respective composed steps. After that, all behaviors stay the same such as pre, post steps, etc.

Eample

A composite action:

name: Yarn
inputs:
  node-version:
     default: 14
runs:
    - name: Checkout
      uses: actions/checkout@v2
    - name: Setup Node.js ${{ inputs.node-version }}
      uses: actions/setup-node@v2
      with:
        node-version: ${{ inputs.node-version }}
    - name: Get Yarn cache directory
      id: get-yarn-cache-dir
      run: echo "::set-output name=path::$(yarn cache dir)"
    - name: Cache Yarn dependencies
      uses: actions/cache@v2
      with:
        path: ${{ steps.get-yarn-cache-dir.outputs.path }}
        key: "${{ runner.os }}-node-${{ inputs.node-version }}-yarn-\
          ${{ hashFiles('**/yarn.lock') }}"
        restore-keys: |
          ${{ runner.os }}-node-${{ inputs.node-version }}-yarn-
    - run: yarn install --frozen-lockfile

A repo's workflow using the action:

jobs:
  check-style:
     runs-on: ubunto-latest
     steps:
       - uses: <point-to-yarn-action>
       - run: yarn style:check
  lint:
     runs-on: ubunto-latest
     steps:
       - uses: <point-to-yarn-action>
       - run: yarn lint
  build:
     runs-on: ubunto-latest
     steps:
       - uses: <point-to-yarn-action>
       - run: yarn build

Before the above workflow runs, it is transpiled into:

jobs:
  check-style:
     runs-on: ubunto-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v2
       - name: Setup Node.js ${{ inputs.node-version }}
         uses: actions/setup-node@v2
         with:
           node-version: ${{ inputs.node-version }}
       - name: Get Yarn cache directory
         id: get-yarn-cache-dir
         run: echo "::set-output name=path::$(yarn cache dir)"
       - name: Cache Yarn dependencies
         uses: actions/cache@v2
         with:
           path: ${{ steps.get-yarn-cache-dir.outputs.path }}
           key: "${{ runner.os }}-node-${{ inputs.node-version }}-yarn-\
             ${{ hashFiles('**/yarn.lock') }}"
           restore-keys: |
             ${{ runner.os }}-node-${{ inputs.node-version }}-yarn-
       - run: yarn install --frozen-lockfile
       - run: yarn style:check
  lint:
     runs-on: ubunto-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v2
       - name: Setup Node.js ${{ inputs.node-version }}
         uses: actions/setup-node@v2
         with:
           node-version: ${{ inputs.node-version }}
       - name: Get Yarn cache directory
         id: get-yarn-cache-dir
         run: echo "::set-output name=path::$(yarn cache dir)"
       - name: Cache Yarn dependencies
         uses: actions/cache@v2
         with:
           path: ${{ steps.get-yarn-cache-dir.outputs.path }}
           key: "${{ runner.os }}-node-${{ inputs.node-version }}-yarn-\
             ${{ hashFiles('**/yarn.lock') }}"
           restore-keys: |
             ${{ runner.os }}-node-${{ inputs.node-version }}-yarn-
       - run: yarn install --frozen-lockfile
       - run: yarn lint
  build:
     runs-on: ubunto-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v2
       - name: Setup Node.js ${{ inputs.node-version }}
         uses: actions/setup-node@v2
         with:
           node-version: ${{ inputs.node-version }}
       - name: Get Yarn cache directory
         id: get-yarn-cache-dir
         run: echo "::set-output name=path::$(yarn cache dir)"
       - name: Cache Yarn dependencies
         uses: actions/cache@v2
         with:
           path: ${{ steps.get-yarn-cache-dir.outputs.path }}
           key: "${{ runner.os }}-node-${{ inputs.node-version }}-yarn-\
             ${{ hashFiles('**/yarn.lock') }}"
           restore-keys: |
             ${{ runner.os }}-node-${{ inputs.node-version }}-yarn-
       - run: yarn install --frozen-lockfile
       - run: yarn build

(Which is what I currently have to do)

Benefits

  • The concept is straightforward to understand and doesn't break current Github Actions behavior.

  • It creates a good conceptual separation between composite actions and standalone actions. When you want all pre, post triggers applied to an action, you write a standalone action, you have all the power of JavaScript and Docker to do complex tasks. When you just want to get rid of irrelevant and duplicated code, you just create a composite action.

Hope it makes sense @ethanchewy!

Please react with a thumbs up if you like the proposal otherwise really love to discuss other ways, because I really want this feature.

Recently made the switch from other CI-tech to Github Actions and I'm surprised how limited the reusability capabilities are. Whether we're talking about Composite Actions, YAML anchors or whatnot, I believe what most developers here are looking for is a straightforward way to reuse the common parts of their workflow definitions. And with straightforward I mean I literally want to cut and paste the common YAML stuff (which is about 80%) in a separate file, include it somehow and "it just works". Yes, I am aware there are other ways to accomplish this, such as rewriting the common parts as shell scripts. But that just reduces the whole Github Actions ecosystem to yet-another-script-runner. I like the Workflow dialect and I like the third-party Github Actions, because they save me time having to write custom "deploy-artifact-x-to-cloud-provider-y" I don't want to write nor maintain.

@MartinDevillers I'm in and definitely intered. Could you mention CI solutions that have good reusability capabililies? ๐Ÿ‘ผ

Also worth to mention that during 2021-Q2 workflow templates will probably fix a lot of your current issues, see github/roadmap#98.

Anyway, I just regret that we can't compose actions in composite actions at the moment. This is the biggest thing I miss from composite action in terms of reusability and I don't know if workflow templates will be an adapted solution regarding that point. We have few months to wait so let's see that in Q3 ๐Ÿ˜‰

@MartinDevillers I'm in and definitely intered. Could you mention CI solutions that have good reusability capabililies?

I worked 2 years using Gitlab CI and it was relatively simple compose pipelines.

Disclaimer: I didn't read the whole thread, so I apologize for any said bs upfront

@shouze as surprising as it may sound with Jenkins + sharedlibs you can fairly easily achieve the reusability of the pipelines across repositories.

We're currently on the process to evaluate what we're going to use in my company and of course github actions is one of these options. I don't know if it will work in a real scenario, but my plan was to build a custom pipeline with typescript that follows the steps that we want, leaving some gaps to be filled somehow in the application repositories.

Another disclaimer: I was the kind of user that was happy with using scripted Jenkins pipelines (I know, it's rare) (I also know, groovy sucks), so I was also very happy when I found out I could write a github action in typescript

@MartinDevillers I'm in and definitely intered. Could you mention CI solutions that have good reusability capabililies? ๐Ÿ‘ผ

Sure, Bitbucket Pipelines has support for YAML Anchors. This is limited to reusability within the same file, but since Bitbucket Pipelines uses one file for all config it works. It's definitely not perfect either, but for the purpose of keeping things DRY it works well enough.

Also worth to mention that during 2021-Q2 workflow templates will probably fix a lot of your current issues, see github/roadmap#98.
Anyway, I just regret that we can't compose actions in composite actions at the moment. This is the biggest thing I miss from composite action in terms of reusability and I don't know if workflow templates will be an adapted solution regarding that point. We have few months to wait so let's see that in Q3 ๐Ÿ˜‰

The part about the workflow partials definitely sounds interesting, but the rest sounds a bit uh-- complicated. I don't think I am looking for a "enterprise centrally managed workflow template". I just want to reuse a piece of YAML from one Workflow definition in another. Apologies if I'm oversimplifying stuff or maybe I'm using the wrong approach here (I'm new to Github Actions).

Hi everyone, I recently released an initial version of cdkactions, which is a cdk for GitHub Actions. cdkactions allows you to define GitHub Action jobs & workflows within TypeScript. Since all configuration lives in TypeScript you can also publish those jobs & workflows to npm, allowing you to easily reuse components to fulfill the missing features of current composite actions. It isn't a perfect alternative to fully-functional composite actions, but it gives significantly more flexibility (and typechecking!).

@ArmaanT, if I didn't get it wrong, cdkactions is a YAML generator. That is, it's an offline tool, isn't it?

@umarcor that's correct. At its core, cdkactions is just a way to represent github actions configuration as TypeScript as well as provide a way to translate that TypeScript into yaml files. However, by doing that you get the benefit of being able to package and publish your configuration to npm which would allow you to create reusable workflows and jobs. If you would like to talk more about it feel free to shoot me an email! I don't want to spam this issue with discussion about an unrelated product.

@ArmaanT, thanks for clarifying. I'm not interested in using TypeScript or npm myself, because all the projects I work with have the CI plumbing written in bash and/or Python. Therefore, git or pip do already fit the purpose of allowing distribution of reusable snippets/generators. Anyway, I wish you the best luck for cdkactions!

Could you please provide an update on the status for "Uses" functionality? The linked experiment is now closed and there has not been any activity on it for quite a while now. Is there another dedicated (and active) thread somewhere for this feature which we can follow?

I would like to know the status of this as well. Inability to reference other actions inside a composite run step action is such a pain.

Found this, don't know if it may help to run actions from custom environment: https://github.com/nektos/act

@TingluoHuang @ericsciple Any idea who at GH could update this thread ? it's been 8 months without any news about future of composite steps and nothing related to that in the public roadmap :-/

Thanks a lot <3

@chrispat @thejoebourneidentity from the product team.

I asked @chrispat about this in the AMA that he did about a month ago:
https://github.community/t/community-ama-featuring-github-actions-2pm-pst/158220/19?u=bboles
I am guessing that he won't have an additional information but I would be happy to be wrong.

I created a tool I've called actions-includes which "expands" the composite actions style syntax to allow full access to GitHub action features inside composite steps (including nesting).

You can see how much nicer it makes a quite complicated GitHub actions workflow at https://github.com/SymbiFlow/fasm/blob/0791037622f2e39a3c7dc1b0321d053bbc0b4727/.github/workflows-src/check-install.yml which aims to test that people can successfully install a Python package with C++ extensions in the many different ways people need to. The two major "includes" can be found at;

As this isn't supported by GitHub actions yet, the tool will "flatten" a workflow.yml file with includes by running python -m actions_includes ./.github/workflows-src/workflow-a.yml ./.github/workflows/workflow-a.yml (It will also inject a github action which checks that the expanded workfile is currently up to date and stop the action if it isn't).

It does all this without requiring that includes be available on the machine doing the expanding.

(Note: we do support these attributes being set in workflows for a step that uses a composite run steps action)

Is there an example of this?

@ryan-sf translation: you can use conditional normally except INSIDE the composite steps. so examples are...the standard documentation

I have a lot of parallel jobs inside a workflow that share basically the same stuff except for a step and the name of the job and an if conditional that controls wether the job should be skipped or not.

Does Github plan to enable composite actions on the job level?

I have a lot of parallel jobs inside a workflow that share basically the same stuff except for a step and the name of the job.

@dimisjim for some cases, the matrix feature can handle this. Here's one example:
https://github.com/apache/pulsar/blob/fb605a72ebc38e9a6e2154daedcacebfa8c5fc68/.github/workflows/ci-maven-cache-update.yaml#L47-L69
The additional benefit is that you can run these in parallel and also control the parallelism and "fail fast" behavior.

indeed, but you can't specify a job-level if conditional differently for each one of them?

IMHO you can do what you want with matrix as recommended. you can have optional on a specific step based on the matrix value and you should get the expected behavior (except the name of the job) but is it so important this property ?

@lgmorand That wouldn't work for us because I need to set different jobs.<job_id>.steps[*].if clauses for each job in the matrix, plus there will be a different set of these clauses depending on the job.

I think this is not possible with the matrix strategy right now, no?

you said that "they do the same thing except for a (one?) step", so I understood that one "if" would be sufficient :) So I trust you on this because you know your needs better than me :)

Ok, I edited my initial comment accordingly ๐Ÿ˜‰

A matrix can define a value that can be used in the name of the job. It can also provide a value that can be used in the if: condition for the steps. Depending on the detailed needs, this may be awkward to have 20 steps where only one of them is run in any given job... but it I'm not sure which bit actually can't be done.

https://github.com/twisted/towncrier/blob/master/.github/workflows/ci.yml#L141-L145

        task:
          - name: Flake8
            tox: flake8
          - name: Check Newsfragment
            tox: check-newsfragment

In this case, rather than conditionals I use the tox environment and let tox control what is done. This could as well be used on a step like if: ${{ matrix.task.tox == 'check-newsfragment' }}.

But, maybe I have missed the point.

@altendky What I need is a way to define a function/method with multiple parameters, not just a matrix array.

E.g The value of the if condition is different from the value of the parametrized step, etc.

How's this getting on, is there perhaps a bountysource or something we could chip a quid or two into if this is something we really want?

Would love an update as well. Composing actions from actions would be incredibly useful as it would simplify code for my end-user.

@hross would you have an updates?

I can't give any information about timelines, but we are currently working on composite actions! ๐Ÿš€

We plan on enabling the uses keyword, which allows us to support:

  • Javascript actions (like @actions/checkout)
  • Container actions
  • Nested composite actions (up to a recursion limit)
  • The pre and post steps these actions generate

I'll post periodic updates in this issue as we make progress.

We have an ADR discussing more technical details as well. We welcome any feedback you wish to provide, particularly around use cases/features that would be helpful to you if they aren't covered above!

Please post that feedback in the ADR!

hi @thboop where should we leave the feedback? Here or in the adr?

I see that if is not planned to be supported. This could be helpful for us. for example if i have two workflows which are nearly identical but have a big chunk of identical steps. part of the steps contain if clauses. Now i can't unite them in one action and call this action from both workflows. If the if would be supported i could do it. I know that i can use a javascript action but why would i need javascript if it is just a number of bash steps.

I also see that pre and post steps of composite actions are not supported. I have an important use case. The google cli needs a service account json. I have this json stored in a secret. so i want to write a composite action which writes a secret to a file and a post step where the file is deleted. This is not possible without post build step.

@david-gang, thanks for the detailed feedback! I've made a note of it. Please leave any additional feedback in the ADR. I've updated the above note to make it more clear.

It might be worth having a play with my actions-includes to see how includes can be used in interesting ways.

The uses keyword is now live on Github.com for composite actions ๐ŸŽ‰ ๐ŸŽ‰ . GHES and GHAE support will come in future releases.

Docs and the changelog will be live in the near future, but I know a lot of you have been really excited for this feature, so I wanted to give you a little bit of time to preview it before then!

You can now specify containers, other composite actions (up to a depth of 9) and node actions in additional to the previously available run steps.

For example, maybe you want to reuse steps for your node project:

action.yaml

name: 'Composite Basic Node'
description: 'Composite Basic Node '
inputs:
  version: # determines what node version to install
    required: false
    description: 'input description here'
    default: '12'
outputs: {}
runs:
    using: "composite"
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: ${{inputs.version}}
      - run: npm test
         shell: bash

Or maybe you want to see the full list of ways you can reference actions and steps inside an action.yaml

runs:
  using: "composite"
  steps:
    # Reference a specific commit
    - uses: actions/checkout@a81bbbf8298c0fa03ea29cdc473d45769f953675
    # Reference the major version of a release
    - uses: actions/checkout@v2
    # Reference a specific version
    - uses: actions/checkout@v2.2.0
    # Reference a branch
    - uses: actions/checkout@main
    # References a subdirectory in a public GitHub repository at a specific branch, ref, or SHA
    - uses: actions/aws/ec2@main
    # References a local action
    - uses: ./.github/actions/my-action
    
    # linux only
    # References a docker public registry action
    - uses: docker://gcr.io/cloud-builders/gradle
    # Reference a docker image published on docker hub
    - uses: docker://alpine:3.8
 
    # and run steps!
    - run: echo "hello world"
      shell: bash

We have some feedback items already that will be going live in a runner release in the near future:

We also have some enhancements that we want to get to in the next stage of work:

If you find a bug, have a question, or want to provide feedback for a new enhancement related to composite actions, please open an issue in this repo and tag me!

@thboop this sounds awesome! It's not immediately clear to me what I need to do to achieve the following:

i want a workflow file in a local repo to be able to contain nothing but minimal config that points to a set of steps in another repo, that can receive params/input. That way, I can have my 300 repos point at a single central repo, and only pass in the things that vary across those repos.

@thboop here you explicitly mentioned public. Is it possible to use this feature with private repositories? if yes, how?

    # References a subdirectory in a public GitHub repository at a specific branch, ref, or SHA
    - uses: actions/aws/ec2@main

edit:

ok, I assume one can just clone the action code before trying to use it.

@thboop here you explicitly mentioned public. Is it possible to use this feature with private repositories? if yes, how?

    # References a subdirectory in a public GitHub repository at a specific branch, ref, or SHA
    - uses: actions/aws/ec2@main

edit:

ok, I assuome one can just clone the action code before trying to use it.

Hey @renannprado , this should work for private repositories. You will need to pass the secret into the composite action as an input, and reference it uses the inputs syntax.

i want a workflow file in a local repo to be able to contain nothing but minimal config that points to a set of steps in another repo, that can receive params/input. That way, I can have my 300 repos point at a single central repo, and only pass in the things that vary across those repos.

You should be able to pass inputs into the composite action, and use those in the steps. A great example can be found here: https://docs.github.com/en/actions/creating-actions/creating-a-composite-run-steps-action#creating-an-action-metadata-file

If this isn't want you mean, please open an issue, lets take this discussion to another thread, but based on what you are describing it should be possible with composite actions!

@thboop cool! My assumption was right! I suppose in this case even though you're passing a secret as input, github actions will still redact the secrets from the logs, right?

Closing out this issue now that uses and with are live. We have a separate followup for the if conditional and we can open other issues for the other items talked about in this issue based on feedback.

Thanks so much for the input everyone!

THIS IS HUGE!!! Thank you so much!! can't wait to try it!

Great news!

I just tried it out in one of our workflows and it works great! Thank you for this @thboop.

@thboop and the other fine people working at GitHub: THANK YOU!

This is a significant improvement of the GitHub Actions product. You guys are really on the right path here.

Good job, keep calm and code on!

Folks at GitHub are truly amazing! Was waiting for this to land and I'm so happy it's finally there โค๏ธ ๐Ÿš€

This was a big pain point in using Power Platform ALM. Thank you for fixing this. ๐Ÿ™โ™ฅ๏ธ

@rajyraman I'm (just) curious, how the uses keyword within composite actions impacted usage with PP ?

@lgmorand - It was not possible to port AzDO templates to Actions because of this gap. microsoft/coe-starter-kit#47

Nice!
But sadly only a small step closer, still pretty much unusable for enterprises until private actions are supported.

unusable for enterprises until private actions are supported.

I work at an enterprise and we've been using private actions for ages. Checkout the source to the action, run it, and you're done.

- uses: actions/checkout@v2
  with:
    repository: org/private-action
    path: path/to/your/local/action
    token: or ssh-key to access private action
- uses: ./.path/to/your/local/action

unusable for enterprises until private actions are supported.

I work at an enterprise and we've been using private actions for ages. Checkout the source to the action, run it, and you're done.

- uses: actions/checkout@v2
  with:
    repository: org/private-action
    path: path/to/your/local/action
    token: or ssh-key to access private action
- uses: ./.path/to/your/local/action

Yes. But it is painful. Why nedring to deal with tokens for an internal scoped repo?

this seems to still be unusable in enterprise. I've tried what @fearphage suggested above but it still seems actions can't use secrets passed to the composite job.

No action can use secrets directly, you have to pass them in as inputs.

this seems to still be unusable in enterprise. I've tried what @fearphage suggested above but it still seems actions can't use secrets passed to the composite job.

You will need to pass the secret as an input, please see the example here show casing how to use an input

@TastyPi and @thboop I've tried this exact thing. I get this error where ever I use the secret.
System.ArgumentException: Unexpected type '' encountered while reading 'action manifest root'. The type 'MappingToken' was expected. at GitHub.DistributedTask.ObjectTemplating.Tokens.TemplateTokenExtensions.AssertMapping(TemplateToken value, String objectDescription) at GitHub.Runner.Worker.ActionManifestManager.Load(IExecutionContext executionContext, String manifestFile)

I should perhaps create a new issue, instead of hashing out my issue here?

I should perhaps create a new issue, instead of hashing out my issue here?

Yes, you should. FYI if you can show a portion of your workflow (privat gist on https://gist.github.com/ for instance) that would be a good place to debug and correct the way you're using it.

Closing out this issue now that uses and with are live. We have a separate followup for the if conditional and we can open other issues for the other items talked about in this issue based on feedback.

Thanks so much for the input everyone!

@thboop

Can you comment with the runner version in which this is included please?

Edit: NVM already there: https://github.com/actions/runner/releases since v2.280.x

@thboop Is there a separate ticket for continue-on-error & timeout-minutes support for steps in a composite action?
Or has support already been added?

still pretty much unusable for enterprises until private actions are supported.

Is this issue from the public GitHub roadmap related?
If yes, then we will see it pretty soon as it's planned for Q4 2021. Really looking forward!

@thboop Any idea when the uses keyword will be available for use in composite actions for GHES? I was looking at the roadmap, but did not see it.

@thboop Will it be possible to use secrets directly in the composite action instead of passing them as input parameters? Or is it not planned to be an option?

I really like the concept of composite actions, but there is a fundamental problem with composite actions and templates which are supposed to be used in the same repository. Let's look how it is currently done, to re-use composite actions, which are located in the very same repository.

    # References a local action
    - uses: ./.github/actions/my-action

So far so good, BUT:

In our CI / CD workflow, we check out the repository only once in the very first job. We do that, because the checkout takes some time, as our mono repository is quite large. From this initial checkout, we then pass relevant parts of our repository to the corresponding, following jobs as artifacts. And it works fine and makes our CI / CD architecture much faster.

But the issue with that, is that I basically can't use template actions from the repositories .github directory in all the following jobs anymore. The workaround would be, to also upload the .github directory as an artifact and to pass them to all following jobs as well, but I don't really like this approach, because it creates so much more boilerplate definitions/steps in the YAML file.

An additional option like

    # References a local action
    - uses: local-github-workflows@./.github/path/to/my-action

would really help to solve this problem. It would be basically a sparse checkout, without doing all this overhead definitions to make them re-usable in every job. In fact, my expectation was, that the .github directory is always accessible from all jobs, even if there is no checkout at all, as it's more of a meta definition for the pipeline and not part of the actual source code.

Either this is already possible, and I am doing something wrong, or this is really something, which needs to be supported as well. I would love to find a better approach for handling templates, as I'm right now very annoyed with this.

@aramfe, what about "other jobs" consuming the action as if it was external?

- uses: your/repository/.github/path/to/your-action@main

That is uncomfortable in the PRs where the action itself is modified. However, it should work for any other modification.

Overall, I agree with your feature request, as I'm running into a similar use case. I would like to include the actions/checkout step inside the reusable workflow (composite action). For completeness, see https://github.com/hdl/containers/blob/main/.github/workflows/boolector.yml#L57-L68 and https://github.com/hdl/containers/blob/main/utils/build-test-release/action.yml#L55-L62. Using it as external does not work in this case, because the Action is expected to be modified frequently.

pllim commented

Is the composite action mentioned in #646 (comment) the same thing as re-usable workflows mentioned in https://docs.github.com/en/actions/learn-github-actions/reusing-workflows ?