Dependencies between orb jobs

[matfax]

I am facing a discrepant challenge developing orbs. On the one hand, you want to have a good separation of concerns, i.e., many separate workflow jobs such as in orb-tools/pack and orb-tools/publish. On the other hand, you want to make the orb usage as simplistic and idiomatic as possible. A simplistic solution is to provide only single inherent jobs that do not depend on one another. That, however, trades off the benefits of multiple workflow jobs. In contrast, if you provide multiple jobs, the schema does not validate the order or dependency chain. The user has to follow the documentation on how to properly use the desired orb.

Dependency Validation

A first approach to solve this would be a dependency validation such as shown in the following example based on the orb-tools:

jobs:
  publish:
    description: >
      Uses the CLI to publish an orb to the registry.
    parameters:
      ...
    steps:
      ...
    requires:
      - pack

That way, the dependency is explicitly traceable for the user and also the validator. If the user of the orb doesn't define a orb-tools/pack job in the workflow that the specified orb-tools/publish requires, the config validation will fail.

Passing:

workflows:
  publish:
    jobs:
      - orb-tools/pack:
          name: my-pack
          ...
      - orb-tools/publish:
          ...
          requries:
            - my-pack

Failing:

workflows:
  publish:
    jobs:
      - orb-tools/pack:
          name: my-pack
          ...
      - orb-tools/publish:
          ...

Error ~ orb-tools/publish requires another job orb-tools/pack

That would be particularly useful for inter-orb dependencies. For instance, if one orb job requires another orb's job first.

jobs:
  my-job:
    requires:
      # This my-job version requires any orb-tools version from 0.1.1 to 0.1.n
      - circleci/orb-tools@0.1.1+/publish

Alternatively, this could also be included in the orbs section:

orbs:
  # If this orb's commands are imported, version 0.1.n is used
  # The user can use any version from 0.1.1 to 0.1.n, if a dependency is defined on this orb
  orb-tools: "circleci/orb-tools@0.1.1+"
jobs:
  my-job:
    requires:
      - orb-tools/publish

Dependency Import

One level further, the required dependencies could also be imported implicitly, if not defined.

jobs:
  publish:
    requires:
      - pack:
           # Pass the parameters as arguments from the publish job to the pack job.
           path: << parameters.path >>
           ...

Passing:

workflows:
  publish:
    jobs:
      - orb-tools/publish:
          path: src/
          ...

The required orb-tools/pack is not defined and hence implicitly added to the workflow.

Processing Result:

workflows:
  publish:
    jobs:
      - orb-tools/pack:
          path: src/
          ...
      - orb-tools/publish:
          ...
          requires:
            - orb-tools/pack

Once you have a dependency chain defined, you can also go further and deduplicate semantically identical deterministic jobs (checking the parameters, determinism must be specified though), and solve the ongoing request for job skipping (1, 2), since jobs need to be rescheduled if any depending jobs rerun.

This concept is just an idea, I felt to share, although I know that is is unlikely to be adopted.

[ndintenfass]

In general, it's true that we're not likely to adopt an in-job dependency declaration. Jobs should be built to stand alone, with any short-cutting when used in tandem baked into the logic. For instance, a parameter that lets you checkout or use a workspace is better than one that assumes checkout or assumes use of an upstream workspace -- that way the job could be used stand-alone, doing a full checkout of your code, or it would be used in a chain, in which case it would use the workspace. That said, we are looking at some kind of shared state mechanism across jobs in an orb and/or better ways to user outputs/artifacts of upstream jobs in a workflow, both of which would be an alternate way to solve for needing jobs to be able to "talk" to each other.