bazooka-ci/bazooka

Support job pipelining

Opened this issue · 4 comments

jawher commented

Make it possible for a job to trigger other jobs after it's finished.

  • A job can list it's downstream jobs, but not the other way around
  • A downstream job is identified by it's name
  • It should be possible for specify the scm revision for a downstream job
  • It should be possible to specify job parameters for a downstream job (#129)
  • For the parameter passing, it'll be much more useful if we could use dynamic values (environment variables for example) in the yaml (#159)
  • A downstream job must be able to access the artifacts of the previous job

An example:

  • api is a bazooka project for a REST server written in Java
  • deploy-api deploys api to a tomcat server
  • api-perf is another project which benchmarks the performance of api
  • api-it runs integration tests on api
  • ui is an angularjs frontend
  • ui-selenium is the last project which tests the ui screens

ui config:

language: java
jdk: 1.8
archive_success: target/api.war
downstream_success:
  - project: deploy-api
    rev: master
  - project: ui
    rev: master

on success, api job triggers deploy-api and ui.

deploy-api config (which supposes the previous job artifacts are mounted on the /incoming directory:

image: debian:jessie
script:
  - scp /incoming/api.war tomcat-admin:$pwd@staging.acme.intranet:/opt/tomcat/webapps/
  - <restart tomcat>
  - <wait a bit>
downstream_success:
  - project: api-perf
    rev: master
  - project: api-it
    rev: master

etc.

jawher commented

Note that this proposed solution is a poor man's pipeline, as it doesn't support some advanced features of a real pipeline with stages like fanning-in (wait for multiple jobs before continuing).

julienvey commented

What about removing pipelines from the yaml file ?

Each bazooka project would have its build configuration in the yaml file, but not the downstream/upstream dependencies. And we implement the pipelines in the Bazooka API.

With this solution, I think it would remove possible errors, such as

  • What happen if the downstream project does not exists
  • If I renamed it ?

And it would also allow us to implement more complex pipelines, fanning-in, human validation...

For simplification, we could consider each downstream jobs gets the artifacts generated by its upstream job in /incoming.

julienvey commented

A list of useful features for pipelining

  • Non sequential logic
    • Parallel
    • Fork/Join
    • Loops
    • Try/Catch/Finally
  • Timeout
  • Retry
  • Human interactions
  • Restartable builds

We only need to start with sequential workflow (downstream/upstream) and create issues for the other features

A pipeline could be described in a yaml DSL. I will try to add a simple example soon

julienvey commented

Just an idea

entry_point: 
  - name: api
    bzk_project: api-unit   # (optional) bazooka project name (if different of name)
    triggered_by: scm  # (optional) scm, manual... defaults to scm
    triggers: 
      -  api-perf
      -  api-it
wait_for:
  - wait_jobs: 
      - api-perf
      - api-it
    triggers: 
      - deploy-api