SSL cert not updating
tactmaster opened this issue · 47 comments
Problem/Motivation
SSL Certs not getting update from lets encrypt
Expected behavior
Lets encrypt ssl cert being automatically updated
Actual behavior
ssl certs not being updated. Is you press renew get this error
Steps to reproduce
Proposed changes
I had the same issue today.
It turned out, that my Pi got a new IP address and my port forwarding pointed to the wrong one.
I had to login to the Pi via SSH and open "/data/logs/letsencrypt/letsencrypt.log" inside the proxymanager docker container to find the exact issue was:
Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
This error I got
Some of certs are renewing some are not it is odd
Command: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-2" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation
[08/Feb/2023:21:15:18 +0100] - 200 200 - GET https aaa.bbb.com "/api/config" [Client 10.0.0.153] [Length 6152] [Gzip -] [Sent-to 192.168.1.75] "-" "-"
[2/8/2023] [9:15:19 PM] [Express ] › ⚠ warning Command failed: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-2" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation
Another instance of Certbot is already running.
Things I tried.
Resetting the database
Uninstalling and restoring from backup.
Deleting the certificate and asking for it again from the UI worked
I'm getting the same "Internal Error" simply when trying to create a brand new SSL cert. I was also getting the error "Another instance of Certbot is already running". I just rebooted my system to get it to shut down all processes... as obviously there was a stuck Certbot instance. After a restart, I wasn't getting the same error... Now I am just getting "Some challenges have failed" from Let's Encrypt and it's unable to generate a new certificate for me. I'm at a loss. If I use Let's Encrypt via the DuckDNS add on, it generates the SSL certs for me just fine and I am then able to reference them via NPM by adding them as "Custom" SSL certs. This works, but it's obviously not ideal because I then have to manually renew the certs anytime they come up for renewal.
DuckDNS add on, it generates the SSL certs for me just fine and I am then able to reference them via NPM by adding them as "Custom" SSL certs. This works, but it's obviously not ideal because I then have to manually renew the certs anytime they come up for renewal.
Would you mind sharing your process? I have the DuckDNS add on but I don't know where it stores the certs so I can add them as custom. My cert expired and I want to keep it going until it hopefully gets patched.
I may be digging myself into a bigger hole tried to add homeassistant.something.duckdns.org
ended up with ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"Incorrect TXT record "dns-01" found at _acme-challenge.homeassistant.something.duckdns.org","status":403}
I noticed in duckdns documentation the folder \ssl\ is supposed to contain the certificates but they are dated December and I tried to just install those anyways and got an error about using those certificates so I don't think those are the right ones.
EDIT
Problem in chair.
Still not 100% sure what fixed it, but I opened up my firewall and reverted the port number for port 80 to port 80.
Also experiencing this issue.
Addon Version: 0.12.3
Upon attempting to create or renew a cert:
[4/10/2023] [3:23:18 AM] [SSL ] › ℹ info Testing http challenge for DOMAIN.REDACTED
Uncaught SyntaxError: Unexpected end of JSON input
FROM
2023/04/10 03:23:19 [error] 298#298: *3 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.4, server: nginxproxymanager, request: "GET /api/nginx/certificates/test-http?domains=%5B%22DOMAIN.REDACTED%22%5D HTTP/1.1", upstream: "http://127.0.0.1:3000/nginx/certificates/test-http?domains=%5B%22DOMAIN.REDACTED%22%5D", host: "10.10.0.200:81", referrer: "http://10.10.0.200:81/nginx/certificates"
[03:23:20] INFO: Nginx Proxy Manager stopped, restarting...
[03:23:20] INFO: Starting the Manager...
Following this crash and restart…
[4/10/2023] [3:23:34 AM] [Nginx ] › ℹ info Reloading Nginx
[4/10/2023] [3:23:39 AM] [SSL ] › ℹ info Requesting Let'sEncrypt certificates for Cert #2: DOMAIN.REDACTED
[4/10/2023] [3:23:39 AM] [SSL ] › ℹ info Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-2" --agree-tos --authenticator webroot --email "EMAIL@REDACTED" --preferred-challenges "dns,http" --domains "DOMAIN.REDACTED"
[4/10/2023] [3:23:44 AM] [Nginx ] › ℹ info Reloading Nginx
[4/10/2023] [3:23:44 AM] [Express ] › ⚠ warning Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-2" --agree-tos --authenticator webroot --email "EMAIL@REDACTED" --preferred-challenges "dns,http" --domains "DOMAIN.REDACTED"
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/certbot-log-j7u3fnai/log or re-run Certbot with -v for more details.
[4/10/2023] [3:24:05 AM] [SSL ] › ✖ error Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
Failed to renew certificate npm-1 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/npm-1/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
at ChildProcess.exithandler (node:child_process:400:12)
at ChildProcess.emit (node:events:513:28)
at maybeClose (node:internal/child_process:1093:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
After this point all attempted Let's Encrypt operations continue to fail.
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
It is still a problem.
I believe that's more of an upstream issue, cf NginxProxyManager/nginx-proxy-manager#2881
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
Absolutely not stale
Do you run a firewall?
It seems like some zone protection or other filters may block these requests. Strangely they didn't show up on the logs as far as I could tell but running with everything on alert made it work smoothly.
Same issue here but deleting, trying to reissue causes more problems. I should have let them expire on their own, lost a month now...
Error: Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-12" --agree-tos --authenticator webroot --email "anthony.scags@gmail.com" --preferred-challenges "dns,http" --domains "rondemealie.duckdns.org"
Another instance of Certbot is already running.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/certbot-log-kuhouk5g/log or re-run Certbot with -v for more details.
at ChildProcess.exithandler (node:child_process:400:12)
at ChildProcess.emit (node:events:513:28)
at maybeClose (node:internal/child_process:1093:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
Still a problem
just had the same issue. For me the issue was misconfiguration under router: port 80 was not forwarded/open.
Certmanager uses Port 80 to execute the HTTP check. Make sure it is open and forwarding to the correct machine
just had the same issue. For me the issue was misconfiguration under router: port 80 was not forwarded/open.
Certmanager uses Port 80 to execute the HTTP check. Make sure it is open and forwarding to the correct machine
this issue talks about problems when enabling forcing on https, the problem is that http verification is also being forced on 443 sending the whole thing into error
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
still an issue
I have the same issue
I feel kinda dumb coming back to this after all this time having issues... but I think I figured out my problem... I've been using this with a DuckDNS domain and it appears it needs a DNS challenge... the DuckDNS addon for HA handles the challenge with the supplied token, which is why I never saw any issues with that addon getting a valid cert. Basically, the problem boils down to there being poor documentation for both the DuckDNS addon and this NPM addon. It was never clear to me that I NEEDED to do the DNS challenge... but it seems obvious now. Basically, I stopped using the DuckDNS addon... I no longer use a custom cert in NPM. Just have NPM create and manage the certs and use the appropriate DNS challenge for your DDNS provider. This appears to be working fine for me this way. Time will tell if the certs update properly when needed, but they should now that they have the appropriate token.
I feel kinda dumb coming back to this after all this time having issues... but I think I figured out my problem... I've been using this with a DuckDNS domain and it appears it needs a DNS challenge... the DuckDNS addon for HA handles the challenge with the supplied token, which is why I never saw any issues with that addon getting a valid cert. Basically, the problem boils down to there being poor documentation for both the DuckDNS addon and this NPM addon. It was never clear to me that I NEEDED to do the DNS challenge... but it seems obvious now. Basically, I stopped using the DuckDNS addon... I no longer use a custom cert in NPM. Just have NPM create and manage the certs and use the appropriate DNS challenge for your DDNS provider. This appears to be working fine for me this way. Time will tell if the certs update properly when needed, but they should now that they have the appropriate token.
I just tried this. Disdabled DuckDNS, NPM began working when forcing a challenge and using my DuckDNS token
All seems to be ok for the moment!
Thanks for this
this issue talks about problems when enabling forcing on https, the problem is that http verification is also being forced on 443 sending the whole thing into error
Had same Internal Error issue. Turned off the force SSL and renewal worked.
Still an issue with Route53 DNS challenge.
just had the same issue. For me the issue was misconfiguration under router: port 80 was not forwarded/open.
Certmanager uses Port 80 to execute the HTTP check. Make sure it is open and forwarding to the correct machine
This helped me too ... It appears that on the WAN interface it has to be port 80 to work properly.
I came across this while troubleshooting my own certificate issues. A couple things that resolved mine:
-
"Another instance of Certbot is already running" - This probably means you're trying to use something else to update your certs. For me, that was my DuckDNS add-on like mentioned above. Stopping that resolved that specific error for me.
-
If you're trying to do a DNS challenge in order to avoid port forwarding from your firewall, you may need to dig into logs more:
I had to login to the Pi via SSH and open "/data/logs/letsencrypt/letsencrypt.log" inside the proxymanager docker container to find the exact issue
The errors in my log weren't particularly helpful, but it listed the certbot-dns-duckdns plugin being used for my DNS Challenge type. Checking the version of that (installed by pip) showed me that the package was super outdated. Updating that package resolved my issue.
My certbot itself is also outdated, but I didn't need to update it in order to resolve my issue. Should any of these packages be getting updates during the addon updates?
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
Any help here?
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
Yeah, my certs failed to renew this month. Not sure why, I'm hoping to dig into it more today. Hitting the button to manually renew worked on 2 of them, but not on my 3rd.
Do you use the "Force SSL" option in the ssl settings?
This was the cause in my homelab, i was able to fix it with a workaround. (custom force-ssl.conf)
For testing you can disable "Force SSL" temporary.
I have faced with same issue: can create a certificate, but it fails to renew after 2 months, with error like: Failed to renew certificate npm-2 with error: The DNS response does not contain an answer to the question
In my case issue was in the local DNS setup. It has a special cname record for resolve services by NPM domain names in LAN, which uses the ^(.*\.)?mydomain\.duckdns\.org$ regex. The regex picks requests to mydomain.duckdns.org and *.mydomain.duckdns.org wildcard, but Let's Encrypt uses the _acme-challenge.mydomain.duckdns.org to verify domain owner by placing a token to TXT record. Let's Encrypt sends a request from LAN during certificate renewal process, the request goes to the local DNS where TXT record does not have the token to validate owner, since TXT is updated on DuckDNS.
So the fix for me was to exclude the _acme-challenge subdomain from resolving scope of local DNS:
^(?!_acme-challenge)(.*\.)?mydomain\.duckdns\.org$
I have faced with same issue: can create a certificate, but it fails to renew after 2 months, with error like:
Failed to renew certificate npm-2 with error: The DNS response does not contain an answer to the questionIn my case issue was in the local DNS setup. It has a special cname record for resolve services by NPM domain names in LAN, which uses the^(.*\.)?mydomain\.duckdns\.org$regex. The regex picks requests tomydomain.duckdns.organd*.mydomain.duckdns.orgwildcard, but Let's Encrypt uses the_acme-challenge.mydomain.duckdns.orgto verify domain owner by placing a token to TXT record. Let's Encrypt sends a request from LAN during certificate renewal process, the request goes to the local DNS where TXT record does not have the token to validate owner, since TXT is updated on DuckDNS. So the fix for me was to exclude the_acme-challengesubdomain from resolving scope of local DNS:^(?!_acme-challenge)(.*\.)?mydomain\.duckdns\.org$
This is interesting. This seems to have been related to my issue, but I'm not 100% sure how to fix it properly. My DNS rewrite only handles mysub.mydomain.duckdns.org (rather than *.mysub.mydomain.duckdns.org). I have a separate rewrite for mydomain.duckdns.org. My AdGuard Home addon is handling my DNS rewrites. I could see the DNS requests for mysub.mydomain.duckdns.org (no reference to _acme-challenge listed, but clearly labeled as a TXT request). I deleted the mysub.mydomain.duckdns.org rewrite just to test and manually refreshed the cert (which worked).
The more interesting thing is that my cert for mydomain.duckdns.org was able to manually refresh even with its rewrite still in place. All doing the DNS-01 challenge. But I don't even see the mydomain.duckdns.org TXT DNS request in the AdGuard logs.
When I test a ping to <something>.mysub.mydomain.duckdns.org, it doesn't route to my LAN, but rather to my public IP. I also see that specific request, including <something> within AdGuard. It isn't referencing an _acme-challenge sub-sub(?) domain within the AdGuard logs during renewal, and if it did then it shouldn't be routed to my LAN anyway based on my ping testing. Maybe I can change the Type of requests that are rewritten? I'm inexperienced with the custom filtering rules, so it looks like I'll need to do a bit more research into that side of things. Maybe I'm implementing my DNS rewrites poorly, or maybe I'm making totally incorrect assumptions based on my ping testing due to it being a different type of DNS request?
This is interesting. This seems to have been related to my issue, but I'm not 100% sure how to fix it properly. My DNS rewrite only handles
mysub.mydomain.duckdns.org(rather than*.mysub.mydomain.duckdns.org). I have a separate rewrite formydomain.duckdns.org. My AdGuard Home addon is handling my DNS rewrites. I could see the DNS requests formysub.mydomain.duckdns.org(no reference to_acme-challengelisted, but clearly labeled as a TXT request). I deleted themysub.mydomain.duckdns.orgrewrite just to test and manually refreshed the cert (which worked).The more interesting thing is that my cert for
mydomain.duckdns.orgwas able to manually refresh even with its rewrite still in place. All doing the DNS-01 challenge. But I don't even see themydomain.duckdns.orgTXT DNS request in the AdGuard logs.
Here a lot of things to check:
- Make sure that there is no another DNS in use. In my router got 2 additional DNS from ISP which were propagated to all DHCP clients. In result: requests were balanced between AdGuard and ISP DNSs
- Flush DNS cache in AdGuard after updating rules
- Flush DNS cache in router if it is used as proxy for AdGuard
When I test a ping to
<something>.mysub.mydomain.duckdns.org, it doesn't route to my LAN, but rather to my public IP. I also see that specific request, including<something>within AdGuard. It isn't referencing an_acme-challengesub-sub(?) domain within the AdGuard logs during renewal, and if it did then it shouldn't be routed to my LAN anyway based on my ping testing. Maybe I can change theTypeof requests that are rewritten? I'm inexperienced with the custom filtering rules, so it looks like I'll need to do a bit more research into that side of things. Maybe I'm implementing my DNS rewrites poorly, or maybe I'm making totally incorrect assumptions based on my ping testing due to it being a different type of DNS request?
Try to monitor dig mydomain.duckdns.org TXT from lan and internet during renewal.
And as I know renewal in NPM will actually renew certificate only if it older then 2 month, unfortunately i don't know how to force it.
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
mope
For anyone else using AdGuard Home alongside their NPM and having issues with renewing their SSL certs, it might be that your AdGuard Home is doing a DNS rewrite of your TXT requests. You may be able to see this within your AdGuard logs. For me, I didn't see the _acme-challenge subdomain within the logs, but I DID see the Type: TXT. More on DNS-01 challenges
To solve this, I needed to remove the DNS rewrite from the Filters->DNS rewrites section and make a custom rule (Filters->Custom filtering rules). The Syntax for this is a bit annoying if you aren't used to it (like me), but after some trial and error plus reading the docs, I was able to come up with a rule to work for me that I couldn't find elsewhere:
|example.com^$dnsrewrite=1.2.3.4,dnstype=A|CNAME|AAAA
I didn't want to rewrite all subdomains, so the single pipe at the beginning | resolves this. If you want to include all subdomains, I believe you can use a double-pipe ||. I tried following the docs where it says |example.com^$dnsrewrite=NOERROR;A;1.2.3.4 adds an A record with the value 1.2.3.4, thinking it would only rewrite A records, but this wasn't the case and it rewrote all other types to NOERROR. The dnstype Rule Modifier appended on the end did work though!
Since I just upgraded my NPM add-on and applied these rules, I don't know yet if I'll have issues when my certs try to auto-renew in a couple months. Hopefully the combination of the add-on update and fixing my local DNS finally resolves this for me.
To solve this, I needed to remove the DNS rewrite from the
Filters->DNS rewritessection and make a custom rule (Filters->Custom filtering rules). The Syntax for this is a bit annoying if you aren't used to it (like me), but after some trial and error plus reading the docs, I was able to come up with a rule to work for me that I couldn't find elsewhere:
i was able figured out with DNS rewrite and get certificate with following setup:
Just waiting that renewal will go smoothly