HTML Breakage with Base64 Encoding and 'append_text_html' Function
Closed this issue · 15 comments
What Operating System are you seeing this problem on?
Rocky8
What Hardware is this system running?
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
BIOS Vendor ID: Intel(R) Corporation
CPU family: 6
Model: 158
Model name: Intel(R) Xeon(R) E-2286G CPU @ 4.00GHz
BIOS Model name: Intel(R) Xeon(R) E-2286G CPU @ 4.00GHz
Stepping: 10
CPU MHz: 4601.359
CPU max MHz: 4900.0000
CPU min MHz: 800.0000
BogoMIPS: 8016.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0-11
KumoMTA version
2024.05.23-11841789
Did you try the latest release to see if the issue is better (or worse!) than your current version?
Yes, and I updated the version box above to show the version that I tried
Describe the bug
I am trying to integrate an image pixel tracking link into my HTML email content using the following code:
local my_tracking_link = '<img src="http://10.0.0.1/img_tracker.jpg" alt="open tracking pixel">'
msg:append_text_html(my_tracking_link)After appending this image pixel using the append_text_html function and encoding the email as base64, the HTML appears broken when viewed in a Gmail mailbox.
To Reproduce
No response
Configuration
local my_tracking_link = '<img src="http://10.0.0.1/img_tracker.jpg" alt="open tracking pixel">'
msg:append_text_html(my_tracking_link)Expected Behavior
No response
Anything else?
No response
Pls provide the before and after HTML so it's clear how "the HTML appears broken".
Before
<!DOCTYPE html>
<html>
<head>
<title>Sample HTML with Href Link</title>
</head>
<body>
<h1>Sample HTML with Href Link</h1>
<p>This is a sample HTML code with a hyperlink.</p>
<a href="https://www.example.com">Visit Example.com</a>
</body>
</html>**After**
<!DOCTYPE html>=0A <html>=0A <head>=
=0A =0A <title>Sample HTML wi=
th Href Link</title>=0A </head>=0A =
<body>=0A <h1>Sample HTML with Href Li=
nk</h1>=0A <p>This is a sample HTML code with =
a hyperlink.</p>=0A <a href=3D"https://www.e=
xample.com">Visit Example.com</a>=0A =20
<img src=3D"http://10.0.0.1/img_tracker.jpg" alt=3D"open tracking pixel"></=
body>=0A </html>=0A =
=20Note : use encoding - base64
Issue exist only using base64 encoding . there is no issue in normal encoding methods like 'quoted-printable' etc..
Please share the smallest reproduction scenario for this; we need to see what is being injected (the complete message) and what is landing in the inbox (the complete message).
<?php
require 'PHPMailer/src/PHPMailer.php';
require 'PHPMailer/src/SMTP.php';
require 'PHPMailer/src/Exception.php';
use PHPMailer\PHPMailer\Exception;
use PHPMailer\PHPMailer\PHPMailer;
use PHPMailer\PHPMailer\SMTP;
try {
$mail = new PHPMailer(true);
$mail->SMTPDebug = SMTP::DEBUG_SERVER;
$mail->isSMTP();
$mail->Host = 'kumo.host.com';
$mail->SMTPAuth = true;
$mail->Username = 'username';
$mail->Password = 'password';
$mail->SMTPSecure = 'tls';
$mail->AuthType = 'PLAIN';
$mail->Port = 587;
$mail->SMTPOptions = array(
'ssl' => array(
'verify_peer' => false,
'verify_peer_name' => false,
'allow_self_signed' => true,
),
);
$mail->setFrom('info@domain.com', "Sender ");
$mail->addAddress('email@gmail.com', "Recipient Name 1 Sure");
$mail->Subject = "sample email";
$mail->Encoding = 'base64';
$mail->Sender = 'returnninfo@domain.com';
$htmlContent = '<!DOCTYPE html>
<html>
<head>
<title>Sample HTML with Href Link</title>
</head>
<body>
<h1>Sample HTML with Href Link</h1>
<p>This is a sample HTML code with a hyperlink.</p>
<a href="https://www.example.com">Visit Example.com</a>
</body>
</html>
';
$mail->Body = $htmlContent;
$mail->IsHTML(true);
if (!$mail->send()) {
echo 'Mailer Error: ' . $mail->ErrorInfo;
} else {
echo 'Message sent successfully!';
}
} catch (Exception $e) {
echo "Message could not be sent. Mailer Error: {$mail->ErrorInfo}";
}Use above php code to sent emails and use append_text_html function in lua file
local my_tracking_link = '<img src="http://10.0.0.1/img_tracker.jpg" alt="open tracking pixel">'
msg:append_text_html(my_tracking_link)`Issue exist in gmail mailbox
We don't have time/resources to replicate your code and run it. Please share the complete message body produced by the code, before it is injected into kumomta, and the resulting complete message body as it appears in the mailbox.
Before injecting image pixel using kumomta
DKIM-Signature: v=1; a=rsa-sha256; d=domain.com; s=kumomta; c=relaxed/relaxed;
bh=GvTClDQcCi5fNfwzYxQElOfPSJ1EfVLmbGXKh6WR7Ac=;
h=from:to:subject; t=1722229496;
b=LZSDnaCSK5ghfeObR3NdyyJRPOHE+Fnt1+cTzSa9EYn6Qmzh7r4Mc7uiwbDMFi67f7/jnsnlk
kUh0cuVrxVDirb4TR6TUOkvrZ8cTNS0Q6vCGH00PtPAC0pgHTH8WelC07xRD00kMA+aVe/LFysP
1qM04nOJCB1yMzAVg6jMDRY=;
DKIM-Signature: v=1; a=rsa-sha256; d=domain.com; s=kmdk; c=relaxed/relaxed;
bh=GvTClDQcCi5fNfwzYxQElOfPSJ1EfVLmbGXKh6WR7Ac=;
h=from:to:subject; t=1722229496;
b=FSxEqP4Uu+NhW8h/VD9+hHS1ogIKJmrNL3CBIceneXZW3gIiLRHXw/mYlLaCFFGGIUkjiBHIt
gLEz/ju1k98kjMF1UY0SFABJ9krnW8TMmBfVdRPjnXzrwxdm0TJ0p3vW5Db+NendsilCpTvp1T+
cakR5Sabk58iO4e6mIhLlD9Gu97T+025EFDwlhtZ0yKuDpSyEWLYwCmUf4QUK8BVlIlyXJAj8lB
OOV7/J/7FzsmRa9Mx/o1pTXYH+AZiEgB4uwbFI78PqN7QMtpUEZxWhKEBpjfN3suMBHEY42me1G
GnYDjvATEI2buDrMrF5BR+jMV3JWc/6Om/rvIdGPhiaA==;
Received: from development.domain.com (10.0.8.1)
by kumo.domain.com (KumoMTA 10.0.8.1)
with ESMTP id 188049cc4d6811efac760cc47a84106e for <arunkvinamcalicut@domain.com>;
Mon, 29 Jul 2024 05:04:55 +0000
Date: Mon, 29 Jul 2024 05:04:55 +0000
To: Recipient Name 1 Sure <***********@domain.com>, Recipient Name 1 Sure <arunk*****@gmail.com>, Recipient Name 1 <*********@gmail.com.com>
From: Sender <aruninfo@domain.com>
Subject: Arun Transaction open check mail(base64) - 1 with inject
X-Mailer: PHPMailer 6.9.1 (https://github.com/PHPMailer/PHPMailer)
mailercloud-track-opens: true
MIME-Version: 1.0
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: base64
Message-ID: <188049cc4d6811efac760cc47a84106e@domain.com>
X-Tracking-Id: 188049cc4d6811efac760cc47a84106e
X-SentFromServer: 10.0.8.1:58816_18_19
X-campaign-Id: 422_domain.com
[1B blob data]
PCFET0NUWVBFIGh0bWw+CiAgICAgICAgICAgICAgICAgICAgPGh0bWw+CiAgICAgICAgICAgICAg
ICAgICAgPGhlYWQ+CiAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAg
ICAgICA8dGl0bGU+U2FtcGxlIEhUTUwgd2l0aCBIcmVmIExpbms8L3RpdGxlPgogICAgICAgICAg
ICAgICAgICAgICAgICAgIDwvaGVhZD4KICAgICAgICAgICAgICAgICAgICAgICAgICA8Ym9keT4K
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxoMT5TYW1wbGUgSFRNTCB3aXRoIEhyZWYgTGlu
azwvaDE+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwPlRoaXMgaXMgYSBzYW1wbGUg
SFRNTCBjb2RlIHdpdGggYSBoeXBlcmxpbmsuPC9wPgogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIDxhIGhyZWY9Imh0dHBzOi8vd3d3LmV4YW1wbGUuY29tIj5WaXNpdCBFeGFtcGxlLmNv
bTwvYT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2JvZHk+CiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgPC9odG1sPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIA==
[1B blob data]
After injecting image pixel using kumomta
DKIM-Signature: v=1; a=rsa-sha256; d=domain.com; s=kumomta; c=relaxed/relaxed;
bh=GvTClDQcCi5fNfwzYxQElOfPSJ1EfVLmbGXKh6WR7Ac=;
h=from:to:subject; t=1722229496;
b=LZSDnaCSK5ghfeObR3NdyyJRPOHE+Fnt1+cTzSa9EYn6Qmzh7r4Mc7uiwbDMFi67f7/jnsnlk
kUh0cuVrxVDirb4TR6TUOkvrZ8cTNS0Q6vCGH00PtPAC0pgHTH8WelC07xRD00kMA+aVe/LFysP
1qM04nOJCB1yMzAVg6jMDRY=;
DKIM-Signature: v=1; a=rsa-sha256; d=domain.com; s=kmdk; c=relaxed/relaxed;
bh=GvTClDQcCi5fNfwzYxQElOfPSJ1EfVLmbGXKh6WR7Ac=;
h=from:to:subject; t=1722229496;
b=FSxEqP4Uu+NhW8h/VD9+hHS1ogIKJmrNL3CBIceneXZW3gIiLRHXw/mYlLaCFFGGIUkjiBHIt
gLEz/ju1k98kjMF1UY0SFABJ9krnW8TMmBfVdRPjnXzrwxdm0TJ0p3vW5Db+NendsilCpTvp1T+
cakR5Sabk58iO4e6mIhLlD9Gu97T+025EFDwlhtZ0yKuDpSyEWLYwCmUf4QUK8BVlIlyXJAj8lB
OOV7/J/7FzsmRa9Mx/o1pTXYH+AZiEgB4uwbFI78PqN7QMtpUEZxWhKEBpjfN3suMBHEY42me1G
GnYDjvATEI2buDrMrF5BR+jMV3JWc/6Om/rvIdGPhiaA==;
Received: from development.domain.com (10.0.8.1)
by kumo.domain.com (KumoMTA 10.0.8.1)
with ESMTP id 188049cc4d6811efac760cc47a84106e for <arunkvinamcalicut@domain.com>;
Mon, 29 Jul 2024 05:04:55 +0000
Date: Mon, 29 Jul 2024 05:04:55 +0000
To: Recipient Name 1 Sure <arunkvinamcalicut@domain.com>, Recipient Name 1 Sure <arunkvinam@gmail.com>, Recipient Name 1 <arunk@vinamsolutions.com>
From: Sender <aruninfo@domain.com>
Subject: Arun Transaction open check mail(base64) - 1 with inject
X-Mailer: PHPMailer 6.9.1 (https://github.com/PHPMailer/PHPMailer)
mailercloud-track-opens: true
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Message-ID: <188049cc4d6811efac760cc47a84106e@domain.com>
X-Tracking-Id: 188049cc4d6811efac760cc47a84106e
X-SentFromServer: 10.0.8.1:58816_18_19
X-campaign-Id: 422_domain.com
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
[1B blob data]
<!DOCTYPE html>=0A <html>=0A <head>=
=0A =0A <title>Sample HTML wi=
th Href Link</title>=0A </head>=0A =
<body>=0A <h1>Sample HTML with Href Li=
nk</h1>=0A <p>This is a sample HTML code with =
a hyperlink.</p>=0A <a href=3D"https://www.e=
xample.com">Visit Example.com</a>=0A =20
<img src=3D"http://arungotracking.domain.com/tr-opens/MTg4MDQ5Y2M0ZDY4MTF=
lZmFjNzYwY2M0N2E4NDEwNmVfNDIy/blank.gif" style=3D"display:none !important;"=
alt=3D"open tracking pixel"></body>=0A </h=
tml>=0A =20
the content itself looks fine to me, but I noticed that you are signing before modifying the content; that will invalidate the dkim signature. Always sign the message as a the last step!
We are facing a critical issue affecting our email delivery functionality. Despite previous communications, the problem persists and is unrelated to DKIM configurations as initially suspected.
Key points:
1. HTML Content Breakage: When emails are sent to Gmail mailboxes, the HTML formatting is disrupted, making the content unusable.
2. append_text_html Function: This disruption coincides with our use of the append_text_html function in our system.
3. Encoding Conversion: We use base64 encoding for our emails, but when using the append_text_html function, the encoding changes to quoted-printable. This change likely causes the HTML to break.
4. Impact: This issue significantly affects our ability to use Kumomta for sending emails, impacting our operational capabilities and the reliability of our communications.
Additional concerns:
1. Base64 to HTML Encoding: When we send an email with base64 encoding and enable open tracking, the encoding changes to HTML. This is a major concern for us.
2. Gmail HTML Breakage: The output HTML is broken in Gmail mailboxes.
This is a severe issue requiring immediate attention.
Your input message decodes like this:
; base64 -d
PCFET0NUWVBFIGh0bWw+CiAgICAgICAgICAgICAgICAgICAgPGh0bWw+CiAgICAgICAgICAgICAg
ICAgICAgPGhlYWQ+CiAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAg
ICAgICA8dGl0bGU+U2FtcGxlIEhUTUwgd2l0aCBIcmVmIExpbms8L3RpdGxlPgogICAgICAgICAg
ICAgICAgICAgICAgICAgIDwvaGVhZD4KICAgICAgICAgICAgICAgICAgICAgICAgICA8Ym9keT4K
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxoMT5TYW1wbGUgSFRNTCB3aXRoIEhyZWYgTGlu
azwvaDE+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwPlRoaXMgaXMgYSBzYW1wbGUg
SFRNTCBjb2RlIHdpdGggYSBoeXBlcmxpbmsuPC9wPgogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIDxhIGhyZWY9Imh0dHBzOi8vd3d3LmV4YW1wbGUuY29tIj5WaXNpdCBFeGFtcGxlLmNv
bTwvYT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2JvZHk+CiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgPC9odG1sPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIA==
<!DOCTYPE html>
<html>
<head>
<title>Sample HTML with Href Link</title>
</head>
<body>
<h1>Sample HTML with Href Link</h1>
<p>This is a sample HTML code with a hyperlink.</p>
<a href="https://www.example.com">Visit Example.com</a>
</body>
</html>
The output you shared is consistent with a quoted-printable rendition of that same input, with the text you specified inserted before the </body> tag.
The functionality appears to be working as described by the documentation.
What's missing from your report is a more detailed description of the "breakage". What is the practical impact? Are you just concerned that the quoted-printable encoding looks weird? Is that a practical impact to how it renders in the gmail web interface?
Without those details, we are left to assume that the issue is that you are misunderstanding how MIME encoding works.
Thank you for your input. Despite the changes, we are still facing issues with HTML content breakage when emails are sent to Gmail mailboxes.
Key points:
1. HTML Breakage: Emails sent to Gmail have disrupted HTML formatting, rendering the content unusable.
2. append_text_html Function: This issue coincides with using the append_text_html function.
3. Encoding Conversion: Using the append_text_html function changes the encoding from base64 to quoted-printable, likely causing the HTML breakage.
This issue significantly impacts our email delivery and operational capabilities. We urgently need a solution.
The output you shared is consistent with a quoted-printable rendition of that same input, with the text you specified inserted before the tag.
The functionality appears to be working as described by the documentation.
What's missing from your report is a more detailed description of the "breakage". What is the practical impact? Are you just concerned that the quoted-printable encoding looks weird? Is that a practical impact to how it renders in the gmail web interface?
Without those details, we are left to assume that the issue is that you are misunderstanding how MIME encoding works.
Hi @MHillyer
Thank you for your response. As a developer, I’m encoding the email in base64 and expect the final header in the email client to remain in base64. This is a standard practice for developers using systems like Kumomta. It’s important that the encoding is not changed by the MTA, as consistency in encoding is crucial for maintaining the integrity of the email content across different clients.
I appreciate your attention to this matter and look forward to a resolution.
The function you are using operates by decoding the part, updating the data, and then using an implementation defined choice in selecting the encoding to be used for the re-encoded part. There is no requirement to encode with the original transfer encoding because that encoding is, by definition, independent of the decoded content, and has no bearing on how that content will be consumed or displayed.
Please read RFC 2045 Section 6.2 for more information on the semantics of transfer encoding.
The resolution is that there is nothing to fix here; everything is operating as intended and documented.
@wez Thanks for the update. However, the HTML appears broken when the mail reaches the Gmail mailbox. Please check the HTML data below. This is the HTML that is received in the Gmail mailbox, which is the main issue I am facing:
PCFET0NUWVBFIGh0bWw+PTBBICAgICAgICAgICAgICAgICAgICA8aHRtbD49MEEgICAgICAgICAg
ICAgICAgICAgIDxoZWFkPj0NCj0wQSAgICAgICAgICAgICAgICAgICAgICA9MEEgICAgICAgICAg
ICAgICAgICAgICAgICAgIDx0aXRsZT5TYW1wbGUgSFRNTCB3aT0NCnRoIEhyZWYgTGluazwvdGl0
bGU+PTBBICAgICAgICAgICAgICAgICAgICAgICAgICA8L2hlYWQ+PTBBICAgICAgICAgICAgICAg
ID0NCiAgICAgICAgICA8Ym9keT49MEEgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGgxPlNh
bXBsZSBIVE1MIHdpdGggSHJlZiBMaT0NCm5rPC9oMT49MEEgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICA8cD5UaGlzIGlzIGEgc2FtcGxlIEhUTUwgY29kZSB3aXRoID0NCmEgaHlwZXJsaW5r
LjwvcD49MEEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxhIGhyZWY9M0QiaHR0cHM6
Ly93d3cuZT0NCnhhbXBsZS5jb20iPlZpc2l0IEV4YW1wbGUuY29tPC9hPj0wQSAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICA9MjANCjxpbWcgc3JjPTNEImh0dHA6Ly9hcnVuZ290cmFja2lu
Zy5zYWxlc2ppby5jb20vdHItb3BlbnMvWkdWaVlUWTJNakUwWkRreE1URj0NCmxabUkzTkRVd1ky
TTBOMkU0TkRFd05tVmZOREl5L2JsYW5rLmdpZiIgc3R5bGU9M0QiZGlzcGxheTpub25lICFpbXBv
cnRhbnQ7Ij0NCiAgYWx0PTNEIm9wZW4gdHJhY2tpbmcgcGl4ZWwiPjwvYm9keT49MEEgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgIDwvaD0NCnRtbD49MEEgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgPTIwDQo=
<!DOCTYPE html>=0A <html>=0A <head>= =0A =0A <title>Sample HTML wi= th Href Link</title>=0A </head>=0A = <body>=0A <h1>Sample HTML with Href Li= nk</h1>=0A <p>This is a sample HTML code with = a hyperlink.</p>=0A <a href=3D"https://www.e= xample.com">Visit Example.com</a>=0A =20 <img src=3D"http://arungotracking.domain.com/tr-opens/ZGViYTY2MjE0ZDkxMTF= lZmI3NDUwY2M0N2E4NDEwNmVfNDIy/blank.gif" style=3D"display:none !important;"= alt=3D"open tracking pixel"></body>=0A </h= tml>=0A =20
My original HTML is below :
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Sample HTML with Href Link</title>
</head>
<body>
<h1>Sample HTML with Href Link with php</h1>
<p>This is a sample HTML code with a hyperlink.</p>
<a href="https://www.example.com">Visit Example.com</a>
</body>
</html>
the HTML that is received in the Gmail mailbox and my original html is different.