zmcertmgr ERROR: Unable to validate certificate chain
Opened this issue · 22 comments
Been using the script since installing Zimbra a good few years back with no issues. The certificate is now failing to renew on my Ubuntu 16.04 (changed the domain name form the detail below)
checking if mail.mydomain.com expires in less than 29 days
certbot-zimbra v0.7.7 - https://github.com/YetOpen/certbot-zimbra
Checking for dependencies...
Detected Zimbra 8.8.8 on UBUNTU16_64
Using domain mail.mydomain.com (as certificate DN)
Preparing certificates for deployment.
Testing with zmcertmgr.
** Verifying '/run/certbot-zimbra/certs-qGX9ITuN/cert.pem' against '/run/certbot-zimbra/certs-qGX9ITuN/privkey.pem'
Certificate '/run/certbot-zimbra/certs-qGX9ITuN/cert.pem' and private key '/run/certbot-zimbra/certs-qGX9ITuN/privkey.pem' match.
** Verifying '/run/certbot-zimbra/certs-qGX9ITuN/cert.pem' against '/run/certbot-zimbra/certs-qGX9ITuN/zimbra_chain.pem'
ERROR: Unable to validate certificate chain: /run/certbot-zimbra/certs-qGX9ITuN/cert.pem: C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup:unable to get issuer certificate
An error seems to have occurred. Please read the output above for clues and try to rectify the situation.
If you believe this is an error with the script, please file an issue at https://github.com/YetOpen/certbot-zimbra.
Any advise would be great, thanks
This problem coming from new cross chain file.
This problem coming from new cross chain file.
Can you explain a little more?
According to someone on the Lets Encrypt forum
It seems that the application cannot handle the new chain. Let's Encrypt is using a new certificate chain since a few days and it seems that this is giving Zimbra trouble.
Not sure what they have changed and how it would be resolved
For now it will work if you select the old chain on certbot. After September maybe we'll need to upgrade Zimbra's embedded OpenSSL.
Add the preferred-chain parameter on certbot_zimbra.sh:
--- certbot_zimbra.sh 2020-03-24 15:29:13.000000000 -0300
+++ certbot_zimbra2.sh 2021-05-12 09:08:32.172542050 -0300
@@ -445,7 +445,7 @@
"$QUIET" && exec > /dev/null
"$QUIET" && exec 2>/dev/null
# Request our cert
- "$LE_BIN" certonly $LE_PARAMS
+ "$LE_BIN" certonly $LE_PARAMS --preferred-chain 'ISRG Root X1'
e=$?
"$QUIET" && exec > /dev/stdout
"$QUIET" && exec 2> /dev/stderr
Thanks I will get that added to my script although for the that has already run I get
Cert not yet due for renewal
How can I get around this?
Thanks
I ran it with the following flags, with success.
# ./certbot_zimbra2.sh -n -c -e mail.domain1.com.br -e webmail.domain1.com.br
certbot-zimbra v0.7.11 - https://github.com/YetOpen/certbot-zimbra
Checking for dependencies...
Detected Zimbra 8.8.15 on UBUNTU18_64
Using zmhostname to detect domain.
Using domain name1.cloud.domain0.com.br (as certificate DN)
Is this correct? yes
Got 2 domains to use as certificate SANs: mail.domain1.com.br webmail.domain1.com.br
Include these in the certificate? yes
Checking zimbra-proxy is running and enabled
Detecting port from zimbraMailProxyPort
Checking if process is listening on port 80 with name "nginx" user "zimbra"
Nginx templates already patched.
Nginx includes already patched, skipping zmproxy restart.
Detecting certbot version...
certbot 1.15.0
We will now run certbot to request the certificate. Proceed? yes
Running /snap/bin/certbot certonly --webroot -w /opt/zimbra/data/nginx/html --cert-name name1.cloud.domain0.com.br -d name1.cloud.domain0.com.br -d mail.domain1.com.br -d webmail.domain1.com.br
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Cert not yet due for renewal
You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/name1.cloud.domain0.com.br.conf)
What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Keep the existing certificate for now
2: Renew & replace the certificate (may be subject to CA rate limits)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate for name1.cloud.domain0.com.br and 2 more domains
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
Thanks. Not sure when I run it why I get
Incompatible option combination
An error seems to have occurred. Please read the output above for clues and try to rectify the situation.
If you believe this is an error with the script, please file an issue at https://github.com/YetOpen/certbot-zimbra.
I still can;'t get it to work. If you use the example but changing to my domain names .
/certbot_zimbra2.sh -n -c -e mail.domain1.com.br -e webmail.domain1.com.br
Incompatible option combination
If I run it with the -c option to get iti to run it still says even after the change to the script as advised
Preparing certificates for deployment.
Testing with zmcertmgr.
** Verifying '/run/certbot-zimbra/certs-dYhL4FAB/cert.pem' against '/run/certbot-zimbra/certs-dYhL4FAB/privkey.pem'
Certificate '/run/certbot-zimbra/certs-dYhL4FAB/cert.pem' and private key '/run/certbot-zimbra/certs-dYhL4FAB/privkey.pem' match.
** Verifying '/run/certbot-zimbra/certs-dYhL4FAB/cert.pem' against '/run/certbot-zimbra/certs-dYhL4FAB/zimbra_chain.pem'
ERROR: Unable to validate certificate chain: /run/certbot-zimbra/certs-dYhL4FAB/cert.pem: C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup:unable to get issuer certificate
You're using certbot-zimbra v0.7.7, please update to 0.7.11.
Hi Nunesvn
I have updated the version as requested but still get errors
Preparing certificates for deployment.
Testing with zmcertmgr.
** Verifying '/run/certbot-zimbra/certs-5mrsxbik/cert.pem' against '/run/certbot-zimbra/certs-5mrsxbik/privkey.pem'
Certificate '/run/certbot-zimbra/certs-5mrsxbik/cert.pem' and private key '/run/certbot-zimbra/certs-5mrsxbik/privkey.pem' match.
** Verifying '/run/certbot-zimbra/certs-5mrsxbik/cert.pem' against '/run/certbot-zimbra/certs-5mrsxbik/zimbra_chain.pem'
ERROR: Unable to validate certificate chain: /run/certbot-zimbra/certs-5mrsxbik/cert.pem: C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup:unable to get issuer certificate
I have updated the version as requested but still get errors
You still have to apply the patch at this comment to make it work
Hi Maxxer
This was added when updating to the latest script
"$LE_BIN" certonly $LE_PARAMS --preferred-chain 'ISRG Root X1'
From this blog post:
Regardless of the certificate chain being used, ISRG Root X1 will need to be present in the “trust store” of clients using any version of OpenSSL. If not, the certificate will be rejected as untrusted.
Also make sure you're using a recent version of Certbot, as that option was introduced in mid Jan
Ok that may prove difficult as when running it says no longer supported
Your system is not supported by certbot-auto anymore.
certbot-auto and its Certbot installation will no longer receive updates.
You will not receive any bug fixes including those fixing server compatibility
or security problems.
Please visit https://certbot.eff.org/ to check for other alternatives.
certbot 1.9.0
Ok ditch certbot-auto and install certbot from packages, or via snap
I tested it on my machine which also had a deployment failure today. The new ISRG Root X1 is now also an intermediate, which is signed by DST Root CA X3. The old intermediate had just one root, this one has 2. Since the script looks for just one, the top-level root is missing from zimbra_chain.pem, so zmcertmgr fails to verify it (fails to find ISRG Root X1's issuer).
So we need to change the chain-building script to loop until it finds the CA (the intermediate that's signed by itself) instead of just hard-coding 1 or 2 passes.
https://letsencrypt.org/certificates/#intermediates
Having cross-signatures means that each of our RSA intermediates has two certificates representing the same signing key. One is signed by DST Root CA X3 and the other is signed by ISRG Root X1. The easiest way to distinguish the two is by looking at their Issuer field.
So maybe old certificates had the R3 intermediate signed by DST Root CA X3, or Ubuntu changed the version of ISRG Root X1 they ship now. It doesn't really matter, we need to handle all of them anyway.
Ok ditch certbot-auto and install certbot from packages, or via snap
I am strongly considering migrating to acme.sh or another similar ACME client, since certbot now wants to be installed via snap (on older distribution releases, the only way to get an up-to-date certbot, since the distro repos have outdated versions), and I refuse to install snap on my servers.
Just for reference after install Certbot on Ubuntu this is what I now get
Certbot can obtain and install HTTPS/TLS/SSL certificates. By default,
it will attempt to use a webserver both for obtaining and installing the
certificate.
certbot: error: unrecognized arguments: --preferred-chain ISRG Root X1
Error: /usr/bin/certbot exit status 2. Cannot proceed, exiting.
Adding that flag to certbot would only be a temporary workaround anyway, we need to fix the script so that it works for all possible chains. Which should be simple enough by writing a while or for loop around the existing issuer cert finding code.
I'm sorry, looking at it closer I see there are actually 3 certificates in the zimbra_chain.pem and the root is not missing. So letsencrypt's chain.pem already has the 2 intermediates and we don't need any loop.
Now I don't know why zmcertmgr doesn't like that:
- su - zimbra -c '/opt/zimbra/bin/zmcertmgr verifycrt comm /run/certbot-zimbra/certs-SvYuEMZA/privkey.pem /run/certbot-zimbra/certs
-SvYuEMZA/cert.pem /run/certbot-zimbra/certs-SvYuEMZA/zimbra_chain.pem'
zimbra_chain.pem contains all roots of cert.pem (R3, ISRG Root X1 and DST Root CA X3). By all theory, this should work.
Actually zimbra_chain contains:
# cat /run/certbot-zimbra/certs-SvYuEMZA/zimbra_chain.pem|openssl x509 -noout -subject -hash -issuer -issuer_hash
subject= /C=US/O=Let's Encrypt/CN=R3
8d33f237
issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
4042bcee
# tail -n +31 /run/certbot-zimbra/certs-SvYuEMZA/zimbra_chain.pem|openssl x509 -noout -subject -hash -issuer
-issuer_hash
subject= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
4042bcee
issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3
2e5ac55d
# tail -n +62 /run/certbot-zimbra/certs-SvYuEMZA/zimbra_chain.pem|openssl x509 -noout -subject -hash -issuer
-issuer_hash
subject= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
4042bcee
issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
4042bcee
So the script picks the wrong issuer, since chain.pem now contains 2 intermediates, not just one, openssl x509 processes the top-most one instead of the bottom-most one, and finds the self-signed ISRG Root X1 in /etc/ssl/certs , which the chain.pem already contains in the cross-signed variant. So we need to modify the script to not process the issuer_hash of the first certificate in chain.pem, but the last one.
I'll push a fix for this in the coming days as I'm busy right now too.