Certbot Continues Requests Despite Hitting Let's Encrypt Certificate Rate Limit

[grimavatar]

---

---

Upon hitting the Let's Encrypt quota of issuing 5 certificates for the exact set of domains within a span of 168 hours, the Certbot process is rendered unable to generate any new certificate orders until a specific timeframe set by Let's Encrypt. Nevertheless, Certbot persists in transmitting requests, leading to an increase in unnecessary network traffic and potentially causing strain on system resources. This condition warrants optimization to prevent Certbot from making redundant certificate requests once the rate limit is reached, thereby enhancing overall system performance and stability.

---

Comprehensive Strategies to Address Let's Encrypt Rate Limit Issue in Certbot:

1. Rate Limit Check: Implement a check within Certbot to ascertain if the rate limit has been reached before attempting to request a new certificate.

2. Backoff and Retry Logic: Introduce an exponential backoff and retry logic to Certbot, reducing the frequency of requests to Let's Encrypt when the rate limit has been reached.

3. Configurable Rate Limit Alerts: Add a feature in Certbot that alerts the administrator when the rate limit is close to being reached for proactive manual intervention.

4. Rate Limit Documentation: Improve the project's documentation to clearly explain Let's Encrypt's rate limits, helping users understand and potentially adjust their certificate issuance strategies.

5. Adjust Certificate Request Strategy: Consider adjusting the certificate request strategy to prevent hitting the rate limit by grouping multiple domains under fewer certificates or adjusting the timing of certificate requests.

6. Next Eligible Time Retry or Notification: Implement a feature to notify the admin when the next eligible time for certificate issuance arrives, or program Certbot to automatically attempt a new request at this given time.

7. Switching Certificate Provider (Let's Encrypt vs ZeroSSL): Consider switching to ZeroSSL, which offers unlimited certificates without rate limits, and provides a user-friendly web interface for certificate management.

8. Automated Certificate Provider Switch: Implement an automated solution where Let's Encrypt is the primary choice, and the system automatically switches to ZeroSSL when the rate limit is reached on Let's Encrypt.

These solutions aim to mitigate the issue of hitting rate limits, enhance system performance and stability, and provide flexibility in handling SSL certificates. As always, any changes should be thoroughly tested to ensure they do not introduce new issues or conflicts with existing functionality.

[byjg]

Do you mind sharing the docker-compose/docker command you are using to start EasyHAProxy?

[grimavatar]

compose.yaml

services:
  frontend:
    image: "byjg/easy-haproxy:4.3.0"
    container_name: "frontend"
    restart: "always"
    environment:
      - "HAPROXY_STATS_PORT=false"
      - "EASYHAPROXY_DISCOVER=docker"
      - "EASYHAPROXY_LETSENCRYPT_EMAIL=your@domain.com"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./certs_letsencrypt:/certs/letsencrypt"
    depends_on:
      - "backend"
  backend:
    build: "backend"
    container_name: "backend"
    restart: "always"
    labels:
      - "easyhaproxy.backend.host=domain.com"
      - "easyhaproxy.backend.port=80"
      - "easyhaproxy.backend.localport=80"
      - "easyhaproxy.backend.letsencrypt=true"
      - "easyhaproxy.backend.redirect_ssl=true"

---

Structure

.
├── backend
│   └── ...
├── certs_letsencrypt
│   └── ...pem
└── compose.yaml

[byjg]

Let's Encrypt doesn't control the issued certificates. Once the certificate is issue we need to guarantee on our side that the certificate will be available for the apps that will use it and we will renew them once is the time.

Docker Easy HAProxy controls if needs to request a new certificate or not by the existence of this certificate in the folder /certs/letsencrypt inside the container. Since the folder inside the container is destroyed after the container is restarted/stopped we need to persist it in the guest machine by mapping a volume.

As long you keep the contents in the guest machine certs folder intact Docker Easy HAProxy will not request a new certificate.

I did several tests on my side and there is no way to reissue a certificate from EasyProxy if the certificate exists there.

Also, I do another validation based on the "age" of the file to request a renew when the certificate is about to expire. To validate the age I simply check the creation timestamp provided by the OS. I tested this feature only in Linux Machines. I can't tell if guest OS based on Windows or MacOS will behave properly. Also if the Guest Machine has the timestamp disabled, the operation will fail. But for these cases, the operation is renew not issue

[grimavatar]

Just to be on same page, do you consider the issue I posted to be valid?

[byjg]

I do. It might have a specific situation that the algorithm is not working. However, I need to reproduce it to fix something.

To test it, I spun up a server with Let's Encrypt, using the same parameters you are, I stopped, restarted, etc and there was no certificate reissuing.

The domain will be up for some days.
https://portainer.xpto.us

First, I stopped the containers using docker-composer down (stop and remove the containers)

Then I started again:

Now, if, for some reason, the certificate is deleted, then a new certificate is issued:

So, my guess, there is something in your system is causing the certificate reissue. I just need to find what it is.

[grimavatar]

Steps to replicate the problem:

• Exceed the limit of 5 certificates.
• Initialize a fresh EasyHAProxy container, ensuring no connections to any past certificates are present.

[byjg]

OK. Let's try to work on this together.

1. Why is HAProxy getting more than one certificate? As I show, once issue and saved in the guest volume, there will be no new requests to CertBot.

Some possibilities:

• More than one EasyHAProxy instance?
• Destroying and recreating the volume?
• Another other?

[grimavatar]

It appears that we're not understanding each other correctly. Your attention seems to be concentrated on replicating how the limit was achieved. However, my concern pertains to the tendency of EasyHAProxy to persist in sending requests for a new certificate, despite having been notified that the limit has been hit and no further certificates will be issued until a designated date.

Consider it from this perspective: a user has already reached the limit through a different method and now opts to use EasyHAProxy for the first time. In this scenario, the problem that I've described above comes into play.

[byjg]

Gotcha. Sorry for the confusion I created. The issue is not why we hit the rate limit. We just hit the rate limit for any reason. Done. Instead to insistently trying to request something won't work, just pause and apply one or more the strategies you mentioned in the first the post.

I'll check them. I didn't knew ZeroSSL too. I'll check. I'll come up with a couple of solutions soon.

[grimavatar]

I came across ZeroSSL when I was seeking out a proxy server or load balancer, specifically SWAG (you can find more about it here), before settling down with yours :-)

Now that we have a mutual understanding, your initial attempt to determine how the limit was initially reached was indeed heading in the right direction. XD

To keep it concise, EasyHAProxy did generate a certificate for me which worked well initially. A few days later, after some experimentation, including deleting and recreating the container/image, I ran into a problem with the certificate and received an error. I resolved this by removing the certificate, allowing EasyHAProxy to create a new one, and that's when I encountered the issue of this post (#41).

I will dig more into it, and post a new issue once I have a clearer view on it. Altogether, there are possible 2 new issues, both which I will post at later time.

[byjg]

I was taking a look on ZeroSSL. As I could understand so far, to use the ZeroSSL API I have:

• Create an account and get an API Key (Manual Step)
• Create the Certificate (can be through the API)
• Validate the Domain. This step I CAN'T automate, because the user needs change the DNS OR upload a file I cannot generate though the API OR Follow the steps in the email. I have no control over this.
• Download the certificate (can be through the API).

Maybe I can automate only after the first certificate was issued to download the certificates. I don´t know need to test.

Let me focus on stop infinite requests to Let's Encrypt when it cannot be done.

[grimavatar]

1. Have you taken a look at their ACME documentation?

2. A brief review of SWAG shows that a user only needs to supply a domain and email, which is identical to Let's Encrypt. This is why I originally suggested this.

3. A brief Google search led me to zerossl-bot, a Certbot wrapper. The procedure appears to be quite straightforward: sudo zerossl-bot certonly --standalone -m anton@example.com -d mydomain.example.com.

[byjg]

Since ZeroSSL can use the Certbot I was able to adapt what I already had for Let's Encrypt and added ZeroSSL and virtually any other issuer that use ACME and Certbot.

So, as you can see:

😍

I also created a documentation for ZeroSSL as well.
https://github.com/byjg/docker-easy-haproxy/blob/667861a8e9fa721b525fd58e53ce5f6f5439f744/docs/zerossl.md

I need to implement other features in the other PRs, and it will take some time to release a version.

If you want to try locally, execute these steps in the EasyHAProxy server:

git clone git@github.com:byjg/docker-easy-haproxy
cd docker-easy-haproxy
git checkout issue/41
docker build -t byjg/easy-haproxy:local -f build/Dockerfile .

And use the image: byjg/easy-haproxy:local instead.

[grimavatar]

I know I suggested it, but I still take simplicity any day of the week. I hope ZeroSSL doesn't make this slower. Also, what happened to option where you just submit domain and email, personally I don't want to open account or deal with extra baggage.

[byjg]

For Let's Encrypt nothing changes.

However ZeroSSL requires you to create an account there and grab or the API Key or the EAB keys. I opted for the last, since there is no changes in the Certbot issuing.

[grimavatar]

I don't like that approach.

[byjg]

And what approach do you suggest?

[grimavatar]

Where no account needs to be created, and EasyHAProxy just uses the same info it was provided for Let's Encrypt. So it's all happens in the background, nothing to learn or worry about, just a swift switch as a backup, or optional to switch if the user chooses to. This should be first class. Second class then is that other path you suggested where more overhead is added. But, personally, the less the user has to do with a reverse proxy/load balancer, the more focus and time to spend on the app.

[byjg]

I understand that, and it's the main reason I created EasyHAProxy - to make the things easier. The Let's Encrypt process works seamless for the user, but behind the scenes, requires some steps/validations that are really complex if you are implementing a pure HAProxy.

Regarding ZeroSSL, it has your own process to create the certificate, but also supports the ACME Protocol. And according to the documentation, there is no other way to create a certificate if you don't have an account there:
https://zerossl.com/documentation/acme/

The implementation as is supports out of the box numerous SSL Issuers free, for example:

• Let's Encrypt
• Buypass
• ZeroSSL,
• SSL.com
• Google Trust Services

And some other paid like:

• DigiCert
• Entrust
• Sectigo

All of them, except Let's Encrypt, require an account.

And setup 3 additional environment variables to support that number of providers, still part of the Easy HAProxy "mission".

I believe BuyPass can issue free certificate without creating an account, but you still need to provide your email and the endpoint: https://community.buypass.com/t/63d4ay/buypass-go-ssl-endpoints-updated-14-05-2020

SSL.com requires you create and account and generate the EAB. https://www.ssl.com/how-to/order-free-90-day-ssl-tls-certificates-with-acme/

I can't see other option for that.

[grimavatar]

I think I found the caveat (source):

"ZeroSSL optionally requires you to register an account with at ZeroSSL.com... Or you can just pass your email address on command line to register with ZeroSSL and automatically obtain and register your EAB credentials..."

So while indeed you do need an account which sucks, you can get away by automating creation, etc, etc via API.

[byjg]

OK. I followed the instructions there and now ZeroSSL can issue certificate without an account. So, I created a configuration with "autoconfig" that makes "easier" to use some pre-defined CA.

Also, if the issue fails for any reason, I implemented a "pause" to avoid hitting every 10 seconds the server.
https://github.com/byjg/docker-easy-haproxy/blob/4bb1c7931d29a223bc8b080630bdf27f695dde25/docs/acme.md