Charset mismatch in webhook body from Bitbucket Server
KalleOlaviNiemitalo opened this issue · 2 comments
Jenkins and plugins versions report
Environment
Jenkins: 2.414.1
OS: Windows Server 2012 R2 - 6.3
Java: 17.0.8.1 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:4.20.0
analysis-model-api:11.7.0
ansicolor:1.0.4
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
atlassian-bitbucket-server-integration:3.6.0
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.7.1
azure-ad:404.v99a_327a_b_8904
azure-sdk:132.v62b_48eb_6f32f
basic-branch-build-strategies:81.v05e333931c7d
bitbucket-server-checks:1.0-SNAPSHOT (private-49d94041-kalle)
bootstrap5-api:5.3.0-1
bouncycastle-api:2.29
branch-api:2.1122.v09cb_8ea_8a_724
buildtriggerbadge:251.vdf6ef853f3f5
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.1
cloudbees-bitbucket-branch-source:832.v43175a_425ea_6
cloudbees-disk-usage-simple:182.v62ca_0c992a_f3
cloudbees-folder:6.848.ve3b_fd7839a_81
command-launcher:107.v773860566e2e
commons-httpclient3-api:3.1-3
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.10.0-78.v3e7b_ea_d5a_fe1
compress-artifacts:98.vb_20f3c77ddf7
configuration-as-code:1700.v6f448841296e
copyartifact:722.v0662a_9b_e22a_c
credentials:1271.v54b_1c2c6388a_
credentials-binding:631.v861c06d062b_4
data-tables-api:1.13.5-1
display-url-api:2.3.9
dtkit-api:3.0.2
durable-task:523.va_a_22cf15d5e0
echarts-api:5.4.0-5
extended-read-permission:53.v6499940139e5
fast-track:1.0.0
font-awesome-api:6.4.0-2
forensics-api:2.3.0
git:5.2.0
git-client:4.5.0
git-forensics:2.0.0
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
hudson-wsclean-plugin:1.0.8
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:73.vddf737284550
jersey2-api:2.40-1
jira:3.11
job-restrictions:0.8
jquery3-api:3.7.0-1
junit:1217.v4297208a_a_b_ce
lockable-resources:1185.v0c528656ce04
mailer:463.vedf8358e006b_
manage-permission:1.0.1
matrix-auth:3.2
matrix-project:808.v5a_b_5f56d6966
metrics:4.2.18-442.v02e107157925
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
pipeline-build-step:505.v5f0844d8d126
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-graph-view:198.v0844db_6ca_554
pipeline-groovy-lib:687.v62591d623759
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-rest-api:2.33
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
pipeline-stage-view:2.33
pipeline-utility-steps:2.16.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
prism-api:1.29.0-7
resource-disposer:0.23
scm-api:676.v886669a_199a_a_
script-security:1275.v23895f409fb_d
sidebar-link:2.2.4
simple-queue:1.4.3
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:308.ve4497b_ccd8f4
sshd:3.312.v1c601b_c83b_0e
structs:325.vcb_307d2a_2782
timestamper:1.26
token-macro:384.vf35b_f26814ec
trilead-api:2.84.v72119de229b_7
variant:60.v7290fc0eb_b_cd
warnings-ng:10.4.0
workflow-aggregator:596.v8c21c963d92d
workflow-api:1281.vca_5fddb_3fceb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3787.v8f5dcd14a_fa_c
workflow-durable-task-step:1289.v4d3e7b_01546b_
workflow-job:1346.v180a_63f40267
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45
xunit:3.1.3
What Operating System are you using (both controller, and any agents involved in the problem)?
Windows Server 2012 R2
Reproduction steps
Have the Jenkins controller running on Windows.
Create a repository and a "master" branch in Bitbucket Server.
Create a multibranch pipeline project in Jenkins and connect it to the repository via the Bitbucket Branch Source plugin, enabling both branch discovery (all branches) and pull request discovery.
Set up a webhook in Bitbucket Server so that it will notify Jenkins.
Push to Bitbucket Server a branch "test-ä" with a commit that is not on the "master" branch.
Push to Bitbucket Server a branch "test-z" with a different commit that is not on the "master" branch.
Create a pull request from "test-ä" to "master".
Create a pull request from "test-z" to "master".
Expected Results
Jenkins should create four jobs in the multibranch pipeline project, and build each of them:
- for the branch "test-ä"
- for the branch "test-z"
- for the pull request from "test-ä" to "master"
- for the pull request from "test-z" to "master"
Actual Results
Jenkins creates only two jobs in the multibranch pipeline project, and builds them:
- for the branch "test-z"
- for the pull request from "test-z" to "master"
The jobs related to branch "test-ä" are missing.
Then click "Scan Multibranch Pipeline Now" in Jenkins. That makes it create the jobs that were missing.
Anything else?
This seems to be a bug in how bitbucket-branch-source-plugin decodes the webhook request body to a string, here:
IOUtils.toString(InputStream) uses the default charset, which is Windows-1252 on the Windows server where the Jenkins controller runs. The webhook request however has "Content-Type: application/json; charset=utf-8", so the charset does not match.
https://www.rfc-editor.org/rfc/rfc8259#section-8.1 requires JSON data to be encoded in UTF-8, so the plugin should just always use that.
$ git grep -F -w -e IOUtils.toString -- src/main
src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java: String content = IOUtils.toString(response.getEntity().getContent());
src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java: return IOUtils.toString(inputStream, StandardCharsets.UTF_8);
src/main/java/com/cloudbees/jenkins/plugins/bitbucket/hooks/BitbucketSCMSourcePushHookReceiver.java: String body = IOUtils.toString(req.getInputStream());
The second call in BitbucketCloudApiClient was changed to specify UTF-8 in 5dedcda.
Verified that replacing IOUtils.toString(req.getInputStream()) with IOUtils.toString(req.getInputStream(), StandardCharsets.UTF_8) fixes the problem.
Probably, this IOUtils.toString call should likewise be changed to use StandardCharsets.UTF_8:
However, I am not changing that in #763, because:
- That code is not related to webhooks and thus does not belong to this issue.
- That code is only executed if Bitbucket Cloud returns an HTTP status other than 200 OK. The plugin will then throw a BitbucketRequestException in any case. Whether the content in the exception message was correctly decoded does not seem very important.
- I would not be able to test a fix.