eddiewebb/circleci-queue

support for serializing workflows by name

ashanbrown opened this issue · 17 comments

ashanbrown commented

Is your feature request related to a problem? Please describe.
I'd like to serialize a workflow of a specific name without other workflows preventing it from running. Currently we wait for all workflows to complete before starting a workflow.

Describe the solution you'd like
Can we check only for existing workflows by workflow name (similar to what we do for serializing jobs)?

Describe alternatives you've considered
None

Additional context

chabou commented

I need this same feature.
@eddiewebb Would you accept a PR for this?

  • ❤️1
  • 👀1
DReigada commented

I would also like to have this feature.

Would be up for doing the change

eddiewebb commented

@chabou @DReigada

I would happily review pr with this feature. Please include tests that validate.

chabou commented

Can we discuss about the right way to activate/configure this feature?
I was thinking to add an optional parameter like consider-workflow-name (false by default to be backward compatible).

DReigada commented

@chabou I was thinking something similar to that.
But I was wondering how the parameter block-workflow would behave if a new parameter like that is introduced.

Some thoughts (when consider-workflow-name is true):

  • If block-workflow is false it would look for other workflows with the same name as the current one (instead of using the job name)
  • If block-workflow is true, I don't know what the behaviour should be.

It seems that ideally this would be a single parameter with 3 possible values: "block by job name", "block by workflow name", "block by anything".

@eddiewebb Any thoughts how this should be implemented?

chabou commented

I was thinking:

  • block-workflow is true :
    • If consider-workflow-name is true, consider only workflow with same name, or all workflows otherwise
  • block-workflow is false :
    • Ignore consider-workflow-name

This is not ideal but has a better backward compatibility than introducing a multiple values parameter.
I don't know what is the best

  • 👍1
chabou commented

Maybe it could be useful to consider "only in a workflow with the same name" even for job names filtering (ie. block-workflow is false). It would be simple to explain.

  • 👍1
eddiewebb commented
chabou commented

Yes you're right, regex is the most powerful solution.

Deprecatation warning and automatically transforming block-workflow: true into * value for the new parameter seems perfect.

workflow-regex is a perfect name I guess.

Do we agree that this new parameter is used in addition of other filtering parameter like only-on-branch or job-regex?
block-workflow is a parameter but a job too. Should we deprecate this job too and keep only queue/until_front_of_line command with some documentation to explain how to have a first job with this command to block an entire workflow?

chabou commented

If it used in combination with job-regex, should it be considered like an AND or an OR condition?
I think that is a AND: the job will be queued only if there is a job with a name matching job-regex AND in a workflow with a name matching workflow-regex. But in this case, without a job-regex parameter, should workflow-regex disable the default behavior of matching the job name (to behave like the deprecated block-workflow)?

eddiewebb commented
chabou commented

I would suggest these configurations:

  1. regardless of job make all workflows serialize
    job-regex: empty
    workflow-regex: .*
  2. regardless of activity(job) serialize this specific workflow
    job-regex: empty
    workflow-regex: <specific_workflow_name>
  3. regardless of workflow, serialize this job.
    job-regex: <specific_job_name>
    workflow-regex: empty
  4. on a specific branch, serialize this job
    job-regex: <specific_job_name>
    workflow-regex: empty
    only-on-branch: <specific_branch_name>
  1. serialize this job only if it belongs to this specific workflow
    job-regex: <specific_job_name>
    workflow-regex: <specific_workflow_name>

The job block-workflow would be an alias of case 1:
job-regex: empty
workflow-regex: .*

chabou commented
  1. serialize job with the same name (default):
    job-regex: empty
    workflow-regex: empty
valernyx commented

Just chiming in here to offer that we are looking for a feature in our organization to serialize a specific job when it's called in a specific workflow, but not serialize it across all workflows.

We currently have the queue/until_front_of_line enabled for our job that must be serialized but it queues it across all workflows and we only really want to queue it so it can only run one instance of the job within each workflow at a time.

Because we have a large number of workflows, we are getting some very long queues without the ability to do this. I love this orb, and would 100% support a feature to restrict the serialization to within the a workflow, vs. across all. What our ideal state would be is across (5) workflows, only (1) of the job-to-serialize could be running in each workflow at each time, so that it would allow (5) job-to-serialize to run at once, as long as each one was running in a separate workflow.

chabou commented

@valernyx I think your usage is the case 5.
Will it feet your needs to specify job's name and workflow's name in each workflow (five time in your exemple)?

valernyx commented

@chabou Yes, I believe case 5 would support our needs. It sounds like I would be able to specify the job-to-serialize in each workflow and specify the workflow name to serialize it in as well, so as long as it would allow the job to run (1) time in each workflow (while supporting concurrency across workflows), then this sounds perfect 👍

eddiewebb commented

#93 resolved this

  • 🎉1