401 Errors Occur randomly.
sfrycertinia opened this issue · 3 comments
Jenkins and plugins versions report
Environment
Jenkins: 2.346.1
OS: Linux - 4.14.252-131.483.amzn1.x86_64
ace-editor:1.1
amazon-ecs:1.41
ant:475.vf34069fef73c
antisamy-markup-formatter:2.7
apache-httpcomponents-client-4-api:4.5.13-1.0
artifact-manager-s3:633.v4813787e78a_9
authentication-tokens:1.4
authorize-project:1.4.0
aws-credentials:191.vcb_f183ce58b_9
aws-global-configuration:100.v4619b_45d2dfc
aws-java-sdk:1.12.215-339.vdc07efc5320c
aws-java-sdk-cloudformation:1.12.215-339.vdc07efc5320c
aws-java-sdk-codebuild:1.12.215-339.vdc07efc5320c
aws-java-sdk-ec2:1.12.215-339.vdc07efc5320c
aws-java-sdk-ecr:1.12.215-339.vdc07efc5320c
aws-java-sdk-ecs:1.12.215-339.vdc07efc5320c
aws-java-sdk-elasticbeanstalk:1.12.215-339.vdc07efc5320c
aws-java-sdk-iam:1.12.215-339.vdc07efc5320c
aws-java-sdk-logs:1.12.215-339.vdc07efc5320c
aws-java-sdk-minimal:1.12.215-339.vdc07efc5320c
aws-java-sdk-ssm:1.12.215-339.vdc07efc5320c
blueocean:1.25.5
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.25.5
blueocean-commons:1.25.5
blueocean-config:1.25.5
blueocean-core-js:1.25.5
blueocean-dashboard:1.25.5
blueocean-display-url:2.4.1
blueocean-events:1.25.5
blueocean-git-pipeline:1.25.5
blueocean-github-pipeline:1.25.5
blueocean-i18n:1.25.5
blueocean-jwt:1.25.5
blueocean-personalization:1.25.5
blueocean-pipeline-api-impl:1.25.5
blueocean-pipeline-editor:1.25.5
blueocean-pipeline-scm-api:1.25.5
blueocean-rest:1.25.5
blueocean-rest-impl:1.25.5
blueocean-web:1.25.5
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.3-7
bouncycastle-api:2.25
branch-api:2.1046.v0ca_37783ecc5
build-failure-analyzer:2.3.0
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checkmarx:2022.2.3
checks-api:1.7.4
cloudbees-bitbucket-branch-source:773.v4b_9b_005b_562b_
cloudbees-folder:6.729.v2b_9d1a_74d673
command-launcher:1.2
config-file-provider:3.10.0
configuration-as-code:1429.v09b_044a_c93de
configuration-as-code-secret-ssm:1.0.1
copyartifact:1.46.4
credentials:1129.vef26f5df883c
credentials-binding:523.vd859a_4b_122e6
custom-tools-plugin:0.8
display-url-api:2.3.6
docker-commons:1.19
docker-workflow:1.28
durable-task:496.va67c6f9eefa7
echarts-api:5.3.3-1
email-ext:2.88
extended-choice-parameter:346.vd87693c5a_86c
favorite:2.4.1
ffsnsnotifier:1.1
font-awesome-api:6.1.1-1
git:4.11.3
git-client:3.11.0
git-server:1.11
github:1.34.3
github-api:1.303-400.v35c2d8258028
github-branch-source:1637.vd833b_7ca_7654
google-login:1.6
h2-api:1.4.199
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.30
jackson2-api:2.13.3-285.vc03c0256d517
javax-activation-api:1.2.0-3
javax-mail-api:1.6.2-6
jaxb:2.3.6-1
jdk-tool:1.0
jenkins-design-language:1.25.5
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.79
jquery:1.12.4-1
jquery3-api:3.6.0-4
jsch:0.1.55.2
junit:1119.1121.vc43d0fc45561
mailer:414.vcc4c33714601
matrix-auth:3.1.2
matrix-project:771.v574584b_39e60
metrics:4.1.6.2
okhttp-api:4.9.3-105.vb96869f8ac3a
pipeline-build-step:2.18
pipeline-graph-analysis:195.v5812d95a_a_2f9
pipeline-groovy-lib:593.va_a_fc25d520e9
pipeline-input-step:449.v77f0e8b_845c4
pipeline-maven:3.10.0
pipeline-milestone-step:101.vd572fef9d926
pipeline-model-api:2.2097.v33db_b_de764b_e
pipeline-model-definition:2.2097.v33db_b_de764b_e
pipeline-model-extensions:2.2097.v33db_b_de764b_e
pipeline-rest-api:2.24
pipeline-stage-step:293.v200037eefcd5
pipeline-stage-tags-metadata:2.2097.v33db_b_de764b_e
pipeline-utility-steps:2.12.1
plain-credentials:1.8
plugin-util-api:2.17.0
popper-api:1.16.1-2
popper2-api:2.11.5-2
prometheus:2.0.11
pubsub-light:1.16
resource-disposer:0.19
role-strategy:484.v8a_a_e4b_d785fd
scm-api:608.vfa_f971c5a_a_e9
script-security:1175.v4b_d517d6db_f0
skip-notifications-trait:1.0.5
slack:608.v19e3b_44b_b_9ff
snakeyaml-api:1.30.1
sse-gateway:1.25
ssh-credentials:277.v95c2fec1c047
sshd:3.228.v4c9f9e652c86
stashNotifier:1.28
structs:318.va_f3ccb_729b_71
timestamper:1.17
token-macro:293.v283932a_0a_b_49
trilead-api:1.57.v6e90e07157e1
variant:1.4
workflow-api:1164.v760c223ddb_32
workflow-basic-steps:948.v2c72a_091b_b_68
workflow-cps:2725.v7b_c717eb_12ce
workflow-cps-global-lib:581.ve633085a_8a_87
workflow-durable-task-step:1146.v1a_d2e603f929
workflow-job:1186.v8def1a_5f3944
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:625.vd896b_f445a_f8
workflow-support:820.vd1a_6cc65ef33
ws-cleanup:0.42
What Operating System are you using (both controller, and any agents involved in the problem)?
Running using the official Docker images for both controller and agent, running in AWS ECS.
Reproduction steps
- Commit code and wait for build.
Expected Results
Build works and notifies Bitbucket that this has occurred.
Actual Results
Largely works OK.
Less frequently but very regularly receive the following...
Started by upstream project
originally caused by:
Branch event
com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .
HttpResponseProxy{HTTP/1.1 401 [Server: nginx, Date: Wed, 13 Jul 2022 13:18:52 GMT, Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Connection: keep-alive, X-AREQUESTID: , Cache-Control: no-cache, no-transform, Vary: x-ausername,x-auserid,cookie,accept-encoding, WWW-Authenticate: OAuth realm="", X-Content-Type-Options: nosniff] org.apache.http.client.entity.DecompressingEntity@1ff3678a}
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:884)
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getResources(BitbucketServerAPIClient.java:836)
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getServerBranches(BitbucketServerAPIClient.java:589)
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getBranches(BitbucketServerAPIClient.java:580)
at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:820)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:101)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:311)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
Finished: FAILURE
Anything else?
Looking at the code from the stack trace it appears to be loading all the branches of the repo using a paged Bitbucket API. We do indeed see the occurrence of this error as correlated to the number of branches in a repo (Some of ours have many hundred branches). However from a quick glance at the error it appears to be using OAuth, whereas the credentials we have supplied to the plugin are username/password. Not sure if that is relevant/related?
I am also seeing random 401 errors and the only way I can get it working again is to clear a CAPTCHA request in Bitbucket.
I've installed Jenkins from an RPM on a Fedora OS.
I expect to Jenkins to be able to query Bitbucket and retrieve a list of repositories and branches for the given Bitbucket project and then create the necessary jobs.
Instead, I receive the following error:
Started by user
[Tue Oct 04 12:50:42 UTC 2022] Starting organization scan...
[Tue Oct 04 12:50:42 UTC 2022] Updating actions...
Looking up team details of project...
Connecting to https://bitbucket using user/****** (BitBucket username and PAT for user)
ERROR: [Tue Oct 04 12:50:42 UTC 2022] Could not refresh actions for navigator com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator@19f7d03b
com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .
HttpResponseProxy{HTTP/1.1 401 [Date: Tue, 04 Oct 2022 12:50:42 GMT, Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Connection: keep-alive, Set-Cookie: AWSALB=iEN691L4MYfhWZm03u97DS1xKHizDSQ+Q+1Wh/G4vsSuciTdbG04KWs3YWrNiSKmL8wgiIwHOUlIdvKS953cgA3YRNHl+ivejLapY2vhxRQThOFBt9b8teChZreH; Expires=Tue, 11 Oct 2022 12:50:42 GMT; Path=/, Set-Cookie: AWSALBCORS=iEN691L4MYfhWZm03u97DS1xKHizDSQ+Q+1Wh/G4vsSuciTdbG04KWs3YWrNiSKmL8wgiIwHOUlIdvKS953cgA3YRNHl+ivejLapY2vhxRQThOFBt9b8teChZreH; Expires=Tue, 11 Oct 2022 12:50:42 GMT; Path=/; SameSite=None; Secure, X-AREQUESTID: *6HPJ19x770x45931629x0, WWW-Authenticate: Basic realm="Atlassian Bitbucket", vary: accept-encoding] org.apache.http.client.entity.DecompressingEntity@3a3671e0}
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:884)
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:756)
Caused: java.io.IOException: I/O error when accessing URL: /rest/api/1.0/projects/project
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:761)
at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.retrieveActions(BitbucketSCMNavigator.java:539)
at jenkins.scm.api.SCMNavigator.fetchActions(SCMNavigator.java:332)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:487)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:918)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)
[Tue Oct 04 12:50:42 UTC 2022] Consulting Bitbucket Team/Project
Connecting to https://bitbucket using user/****** (BitBucket username and PAT for user)
ERROR: [Tue Oct 04 12:50:42 UTC 2022] Could not fetch sources from navigator com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator@19f7d03b
[Tue Oct 04 12:50:42 UTC 2022] Finished organization scan. Scan took 55 ms
FATAL: Failed to recompute children of PROJECT
com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .
HttpResponseProxy{HTTP/1.1 401 [Date: Tue, 04 Oct 2022 12:50:42 GMT, Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Connection: keep-alive, Set-Cookie: AWSALB=PN9dRjWNalH7fr3ZSCuB365FTcYDIWRt+/c8Jyt+CdicXYFTbT7DlVqPeZ7BvhqnLp2t8gyNqPZhO2tXDM8GtYtQyh6PrAGbG/SaIUJK5sEaentabRO+k5g+Lrm/; Expires=Tue, 11 Oct 2022 12:50:42 GMT; Path=/, Set-Cookie: AWSALBCORS=PN9dRjWNalH7fr3ZSCuB365FTcYDIWRt+/c8Jyt+CdicXYFTbT7DlVqPeZ7BvhqnLp2t8gyNqPZhO2tXDM8GtYtQyh6PrAGbG/SaIUJK5sEaentabRO+k5g+Lrm/; Expires=Tue, 11 Oct 2022 12:50:42 GMT; Path=/; SameSite=None; Secure, X-AREQUESTID: *6HPJ19x770x45931630x0, WWW-Authenticate: Basic realm="Atlassian Bitbucket", vary: accept-encoding] org.apache.http.client.entity.DecompressingEntity@76528557}
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:884)
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:756)
Caused: java.io.IOException: I/O error when accessing URL: /rest/api/1.0/projects/project
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:761)
at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.visitSources(BitbucketSCMNavigator.java:490)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:536)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:918)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)
Finished: FAILURE
This issue also affects builds (not just scanning Bitbucket projects) and the build log reports:
10:53:34 Started by user
10:53:34 com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .
10:53:34 HttpResponseProxy{HTTP/1.1 401 [Date: Tue, 04 Oct 2022 09:53:34 GMT, Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Connection: keep-alive, Set-Cookie: AWSALB=YEUobGH3gNCSWWLlH5sYlFxa7wNhWxDqbH+xzlFH6dwYHIiTKKazmtAQ35xaTiJsXf59fB0JcAi39DcvNSbcskAwUjifJQgVsvDO4xOwgIBWYeFLvc28/ekYH6+u; Expires=Tue, 11 Oct 2022 09:53:34 GMT; Path=/, Set-Cookie: AWSALBCORS=YEUobGH3gNCSWWLlH5sYlFxa7wNhWxDqbH+xzlFH6dwYHIiTKKazmtAQ35xaTiJsXf59fB0JcAi39DcvNSbcskAwUjifJQgVsvDO4xOwgIBWYeFLvc28/ekYH6+u; Expires=Tue, 11 Oct 2022 09:53:34 GMT; Path=/; SameSite=None; Secure, X-AREQUESTID: *Q55LJLx593x46291775x2, WWW-Authenticate: Basic realm="Atlassian Bitbucket", vary: accept-encoding] org.apache.http.client.entity.DecompressingEntity@79f1bdc1}
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:884)
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getResources(BitbucketServerAPIClient.java:836)
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getServerBranches(BitbucketServerAPIClient.java:589)
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getBranches(BitbucketServerAPIClient.java:580)
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:820)
10:53:34 at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
10:53:34 at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:101)
10:53:34 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:311)
10:53:34 at hudson.model.ResourceController.execute(ResourceController.java:107)
10:53:34 at hudson.model.Executor.run(Executor.java:449)
10:53:34 Finished: FAILURE
This is happening increasingly frequently but stills seems somewhat random. I've not been able to correlate certain actions to repeatably cause this.
It seems that no credentials get sent seemingly randomly (~ 1 per 1000 builds?). As we do several thousand builds a day and it seems to be mostly on the same repo (with the most branches) this is quite annoying for the team impacted. Is this just because that makes more requests to get all the branches?
I updated the BitBucket Branch Source plugin for Jenkins (to version 791.vb_eea_a_476405b) a couple weeks ago and since then I've not experienced the 401 / CAPTCHA request issues I described above. So, for me at least, I think it was fixed by #637 (https://issues.jenkins.io/browse/JENKINS-61200).