Multijob infinite recursion when subordinate job is disabled.
Closed this issue · 4 comments
Jenkins and plugins versions report
Environment
Jenkins: 2.332.3
OS: Linux - 5.13.0-44-generic
---
Parameterized-Remote-Trigger:3.1.5.1
ace-editor:1.1
ant:475.vf34069fef73c
antisamy-markup-formatter:2.7
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
bootstrap4-api:4.6.0-5
bootstrap5-api:5.1.3-7
bouncycastle-api:2.26
branch-api:2.1046.v0ca_37783ecc5
build-blocker-plugin:1.7.8
build-name-setter:2.2.0
build-pipeline-plugin:1.5.8
build-publisher:1.22
build-timeout:1.21
build-with-parameters:1.6
buildresult-trigger:0.17
built-on-column:1.1
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
call-remote-job-plugin:1.0.21
checks-api:1.7.4
cloudbees-folder:6.729.v2b_9d1a_74d673
command-launcher:84.v4a_97f2027398
conditional-buildstep:1.4.2
credentials:1087.1089.v2f1b_9a_b_040e4
credentials-binding:523.vd859a_4b_122e6
cvs:2.19.1
dark-theme:156.v6cf16af6f9ef
dashboard-view:2.432.va_712ce35862d
description-setter:1.10
display-url-api:2.3.6
docker-commons:1.19
docker-java-api:3.2.13-37.vf3411c9828b9
docker-plugin:1.2.9
docker-workflow:1.28
doclinks:0.6.1
doxygen:0.18
durable-task:496.va67c6f9eefa7
echarts-api:5.3.2-2
elastic-axis:1.6.0
email-ext:2.88
emailext-template:1.4
embeddable-build-status:2.0.3
envfile:1.2
envinject:2.866.v5c0403e3d4df
envinject-api:1.199.v3ce31253ed13
extended-choice-parameter:346.vd87693c5a_86c
extended-read-permission:3.2
external-monitor-job:191.v363d0d1efdf8
folder-properties:1.2.1
font-awesome-api:6.1.1-1
git:4.11.3
git-client:3.11.0
git-parameter:0.9.16
git-server:1.11
github:1.34.3
github-api:1.303-400.v35c2d8258028
github-branch-source:1637.vd833b_7ca_7654
gitlab-plugin:1.5.33
gradle:1.39
greenballs:1.15.1
groovy:2.4
handlebars:3.0.8
jackson2-api:2.13.3-285.vc03c0256d517
javadoc:217.v905b_86277a_2a_
javax-activation-api:1.2.0-3
javax-mail-api:1.6.2-6
jaxb:2.3.6-1
jdk-tool:1.5
jenkins-multijob-plugin:611.v9d3180d752e6
jersey2-api:2.35-8
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.7-3
job-import-plugin:3.5
job-restrictions:0.8
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.6.0-4
jsch:0.1.55.2
junit:1119.va_a_5e9068da_d7
ldap:2.10
lockable-resources:2.15
mailer:414.vcc4c33714601
mapdb-api:1.0.9.0
matrix-auth:3.1.2
matrix-combinations-parameter:1.3.1
matrix-project:771.v574584b_39e60
maven-plugin:3.19
momentjs:1.1.1
monitor-remote-job:1.0
msbuild:1.30
nodelabelparameter:1.10.3.1
okhttp-api:4.9.3-105.vb96869f8ac3a
p4:1.12.4
pam-auth:1.8
parameterized-trigger:2.44
pipeline-github-lib:38.v445716ea_edda_
pipeline-groovy-lib:591.v3a_7f422b_d058
pipeline-input-step:448.v37cea_9a_10a_70
pipeline-milestone-step:101.vd572fef9d926
pipeline-model-api:2.2086.v12b_420f036e5
pipeline-model-definition:2.2086.v12b_420f036e5
pipeline-model-extensions:2.2086.v12b_420f036e5
pipeline-stage-step:293.v200037eefcd5
pipeline-stage-tags-metadata:2.2086.v12b_420f036e5
plain-credentials:1.8
plugin-usage-plugin:2.2
plugin-util-api:2.17.0
popper-api:1.16.1-3
popper2-api:2.11.5-2
preSCMbuildstep:0.3
rebuild:1.34
remote-jobs-view-plugin:0.0.3
resource-disposer:0.19
role-strategy:488.v0634ce149b_8c
run-condition:1.5
scm-api:608.vfa_f971c5a_a_e9
script-security:1175.v4b_d517d6db_f0
shelve-project-plugin:3.2
snakeyaml-api:1.30.1
ssh:2.6.1
ssh-agent:295.v9ca_a_1c7cc3a_a_
ssh-credentials:277.v95c2fec1c047
ssh-slaves:1.814.vc82988f54b_10
sshd:3.237.v883d165a_c1d3
statusmonitor:1.3
strawboss:1.3
structs:318.va_f3ccb_729b_71
text-finder:1.19
theme-manager:0.6
thinBackup:1.10
timestamper:1.17
token-macro:293.v283932a_0a_b_49
translation:1.16
trilead-api:1.57.v6e90e07157e1
variant:1.4
versioncolumn:2.2
windows-slaves:1.8.1
workflow-api:1164.v760c223ddb_32
workflow-basic-steps:948.v2c72a_091b_b_68
workflow-cps:2725.v7b_c717eb_12ce
workflow-cps-global-lib:588.v576c103a_ff86
workflow-durable-task-step:1144.vd77b_57189936
workflow-job:1186.v8def1a_5f3944
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:625.vd896b_f445a_f8
workflow-support:820.vd1a_6cc65ef33
ws-cleanup:0.42
What Operating System are you using (both controller, and any agents involved in the problem)?
This has been observed for the following scenarios:
| Controller | Agent |
|---|---|
| Linux(ubuntu) | Linux(ubuntu) |
| Linux(ubuntu) | Windows-10 |
Reproduction steps
I've been able to reproduce with a simple dummy scenario:
- Create a disabled freestyle job
- New Freestyle
- Name:01-Disabled
- Description: This is a disabled project
- Build: Add shell or Windows batch command
echo Hello I'm disabled - Disable the job
- Create an an enabled freestyle job
- New Freestyle
- Name:02-Enabled
- Desription:This project is enabled
- Build: Add shell or Windows batch command
echo This step is enabled
- Create a multijob that uses the two
- New Multijob
- Name: 00-Multijob
- Desription: I call 01-Diabled and 02-Enabled
- Build:
- Add multijob phase
- Name: P1
- Job: 01-Disabled
- Continue on Success
- Add multimob phase
- Name: P2
- Job: 02-Enabled
- Continue on Success
- Add multijob phase
- Run the multijob phase
- View the job run information
It goes into infinite recursion in the display
Expected Results
It should not execute the disabled job.
The display should not go into infinite recursion. Previous releases of the plugin would display this as simply 02-Enabled, and not show 01-Disabled at all.
The following screenshot is from our real configuration when it was working (with 1.36). When we have the Test-Linux stage, after Build-Linux disabled. It's disabled in the Test-Linux job, NOT at the multijob layer.
Actual Results
Anything else?
This problem did not start until we upgraded to 611.v9d3180d752e6 from 1.36.
I've attached the config for the simple scenario I've explained.
simple_example.zip
We ran into this as well. The problem gets more noticeable if you allocate a lot of stack memory to the server (-Xss) because it cycles for a very long time and locks up the controller while recursing.
I tried downgrading to 1.36 and experienced the same infinite recursion when displaying the results of an old run that had a disabled job as part of the chain, so it's unclear how new this bug is. I have not figured out whether "new" runs will be healthier.
Our environment is Amazon Linux (redhat'ish) with Jenkins 2.346.3 (recent LTS) running on Java11.
Jenkins (2.414.1)
Multijob plugin:623.v03401733c9a_9
The problem also exists
Issue still exists on plugin version: 627.v7c23cef20a_6a
checked on Jenkins lunched from docker images:
jenkins/jenkins:2.426.2-lts-jdk11
jenkins/jenkins:2.426.2-lts-jdk17
jenkins/jenkins:2.426.2-lts-jdk21
jenkins/jenkins:2.440.3-lts-jdk21
jenkins/jenkins:2.452.3-lts-jdk21
jenkins/jenkins:2.452.3-lts-jdk11
way to reproduce:
multijob_temp is a multijob which triggers tempjob_a and tempjob_b with 20sec of quiet queue
trigger multijob_temp and when tempjob_a is queued then :
- disable sub job tempjob_a when already queued (issue does not occure if job is disabled before trigger multijob_temp)
or
- disable sub job tempjob_a when waitng for assigned node
The behavior is consistent with
- calling table.jelly recursively and
- referencing the multijob itself in SubBuild.buildID property via this call.
I guess in most cases SubBuild is either not created (for the jobs disabled before the run) or created with a proper buildId (for enabled jobs). This was introduced in this change.
We would need to prevent loops in the recursive table.jelly call and maybe rethink buildId.