CacheUnmanagedCertificatePEMBytes returns only the first certificate and omits intermediate and root certificates. What am I doing wrong?

Question

CacheUnmanagedCertificatePEMBytes returns only the first certificate and omits intermediate and root certificates. What am I doing wrong?

Opened this issue 2 months ago · 13 comments

What is your question?

I'm using CacheUnmanagedCertificatePEMBytes and manage custom TLS for several websites using the following code:

cnf := certmagic.NewDefault() 
cnf.CacheUnmanagedCertificatePEMBytes(
  context.Background(),
  []byte(appConfig.CertValue),
  []byte(appConfig.CertKey),
  nil,
)

// I also manage `on-demand` certificates on the same 
// server for applications that do not define a custom certificate:
certmagic.Default.OnDemand = &certmagic.OnDemandConfig{
  DecisionFunc: myDecisionFunc(),
}

// and finally spin up the server using `HTTPS` method:
certmagic.HTTPS(/* ... */)

Despite passing the full certificate chain to certmagic, the server returns only the first certificate and omits intermediary and root certificates.

Browsers are handling this correctly because they retrieve the missing certificates, but when I use SSL tools, they also throw errors.

What have you already tried?

Include any other information or discussion.

Bonus: What do you use this package for, and does it help you?

I use this at stormkit.io to issue dynamic certificates for hosted apps. Self-hosted enterprise customers usually prefer to host their managed certificates so I have to use a combination between dynamic and managed certificates. It works pretty well, thanks for your work 🙏

Answer 1 · 2024-09-09T15:10:49.000Z

Can you attach the full chain file here and I will see if I can reproduce it?

Answer 2 · 2024-09-09T16:43:58.000Z

@mholt there you go:

subject=CN = learn.haj.gov.sa

issuer=C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com RSA SSL subCA

-----BEGIN CERTIFICATE-----
MIIHDjCCBPagAwIBAgIQPstk5XlT2Hmq1k/pat3iwzANBgkqhkiG9w0BAQsFADBp
MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hvdXN0b24x
GDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjEeMBwGA1UEAwwVU1NMLmNvbSBSU0Eg
U1NMIHN1YkNBMB4XDTI0MDkwNTEwMzgzNFoXDTI1MDczMTEwMzgzNFowGzEZMBcG
A1UEAwwQbGVhcm4uaGFqLmdvdi5zYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAJcPIe7BEFEehFLzILVd/46RUPGqTWsKk09olDVGBFLnQ/2ZKEdGSjBz
Cq17BUg2WHRuzvxD6IK+vEseo1evijFPV1r5h55hXcA3VgJHK5cm17Jj4m6zjBQP
IKIYR+aZq3jzJKjvBXy3diDtAV+yoCa43EpIwuoCFw++dlEAGkIwqRaJy0/rd8i1
WS6WVMmY1/69dwm2gP66GAiOe2vI8jtLkWchhMAJkPQEp5P3wWtTreGuorXvlOQz
Aow1F1C2rcddz2k2TWH8uGFbStxYic7W0qflXkBKRENacxqze2SVKK5nrCg7nqI8
gkMhGYgvTjK4viPR4Z7bnP8E8MWo2Z8CAwEAAaOCAv4wggL6MAwGA1UdEwEB/wQC
MAAwHwYDVR0jBBgwFoAUJhR+4NzXpvfi1AQn32HxwuznMsowcgYIKwYBBQUHAQEE
ZjBkMEAGCCsGAQUFBzAChjRodHRwOi8vY2VydC5zc2wuY29tL1NTTGNvbS1TdWJD
QS1TU0wtUlNBLTQwOTYtUjEuY2VyMCAGCCsGAQUFBzABhhRodHRwOi8vb2NzcHMu
c3NsLmNvbTAbBgNVHREEFDASghBsZWFybi5oYWouZ292LnNhMCMGA1UdIAQcMBow
CAYGZ4EMAQIBMA4GDCsGAQQBgqkwAQMBATAdBgNVHSUEFjAUBggrBgEFBQcDAgYI
KwYBBQUHAwEwRQYDVR0fBD4wPDA6oDigNoY0aHR0cDovL2NybHMuc3NsLmNvbS9T
U0xjb20tU3ViQ0EtU1NMLVJTQS00MDk2LVIxLmNybDAdBgNVHQ4EFgQU2hm7/zZO
6QhqcR1AvgOYauM5xgswDgYDVR0PAQH/BAQDAgWgMIIBfAYKKwYBBAHWeQIEAgSC
AWwEggFoAWYAdQAN4fIwK9MNwUBiEgnqVS78R3R8sdfpMO8OQh60fk6qNAAAAZHB
zOPRAAAEAwBGMEQCIHOPq1ZGXAc6bDki3MHw1Q35As6uI0bYCw6ISLecMAzLAiAz
JAHKg/PlRZWZlme7udceeMWoZNDlw5mLR/E+K0SHmgB1AN3cyjSV1+EWBeeVMvrH
n/g9HFDf2wA6FBJ2Ciysu8gqAAABkcHM45YAAAQDAEYwRAIgJd2px98uKBGEvIZN
K3jTOwhJNEog0ZoybnG0XGn5FVcCICylh7nrUHOVT4dGeOxuVhejBWERo44N8h6n
ofGfZv3TAHYAEvFONL1TckyEBhnDjz96E/jntWKHiJxtMAWE6+WGJjoAAAGRwczj
dQAABAMARzBFAiEAlsAEcr3ojgpnpRa5Hx2bOB+F0O8zzUwbeFB12YKCzxACIDjx
JJZovXrkBVOCGvLbi8RBcCrcQCJlbaym1ZBXXeSrMA0GCSqGSIb3DQEBCwUAA4IC
AQBHy/GED47Bbgbuaq+oqYrp1y05QgjRx+nrV8N2bWo9bk8TS1/Bhjph7SrRgYDg
FVRReikBD7EvGd7Oj4osNUpnnn21Njv6snT1xYcpoKGZ2vGmIvqM5Bu46VHrszYT
GPKOt6dU/j4z2BwTffFZssUNiLibAmnDePN39Xns7F8znbQ3meDdwwaT2skqLd8Z
6GQrHAEBP2S90UbT+uWoDykXX2A9taaw+fFXdg3nl6Jx4trJP2oqunrAwCt8ogK0
aiotmm9bnJ8H7mUnip0QqQBAi32p55I0aLUmRue3Y9vc+wCnS7okyftaNGEBfSCo
+bHwRxoSY0S1zGy/BicjKvbMG/7ddZ6aSKPTXCiUsOoPAOeDfB3kVXZ1YTckoHq/
8ukSht2G1eSTOgXnRDXM9N2e9dofNabT8WLpcbV1bbeJ8JUQTnK+kP6r+d575pMv
8gfXZVdUqNCNrIJK0LeaGx4EwAJUFa80rhytE2wqIxO/L5Re3JeoyZoAWgxDynnB
z+l1ybdTL0JsEsei0+UVH6/VWDgRTwwavusiebh4JW5f/lRcFdMJkkXR8o7HeHIR
3z2puzTDSPJNFh8tQgRechKLrhoEKE5O6Ro96NA+9Lw7ztW/kqyqBnsxCqxMHjiq
uEE+i3QuNVFP9v8RCeEf3a50sEDyJPZzbDaHRMuP56Aliw==
-----END CERTIFICATE-----

subject=C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com RSA SSL subCA

issuer=C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com Root Certification Authority RSA

-----BEGIN CERTIFICATE-----
MIIGbzCCBFegAwIBAgIICZftEJ0fB/wwDQYJKoZIhvcNAQELBQAwfDELMAkGA1UE
BhMCVVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQK
DA9TU0wgQ29ycG9yYXRpb24xMTAvBgNVBAMMKFNTTC5jb20gUm9vdCBDZXJ0aWZp
Y2F0aW9uIEF1dGhvcml0eSBSU0EwHhcNMTYwMjEyMTg0ODUyWhcNMzEwMjEyMTg0
ODUyWjBpMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hv
dXN0b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjEeMBwGA1UEAwwVU1NMLmNv
bSBSU0EgU1NMIHN1YkNBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA
hPYpOunhcxiF6xNzl6Tsm/Q89rnu2jVTXTBOZPaBkSD1Ic4lm7qkYwlZ/UgV5nn1
5ohhceYDC2AlR9RvGbP+26qrNcuE0XOdHJOB4SoY4d6OqLAQ6ZB0LdERK1Saa5lp
QlqHE8936dpr3hGWyqMb2LsdUuhQIzwNkLU/n9HO35irKCbKgS3FeejqkdqK5l6B
b11693o4bz9UZCUdBcQ/Xz06tA5cfnHvYkmmjxhj1lLTKwkQhWuIDrpbwWLO0QVO
c29s9ieomRKm8sYMyiBG4QqRQ/+bXwp48cF0qAByGWD6b8/gG4Xq1IBgO5p+aWFS
0mszkk5rsh4b3XbTHohP3oWQIOV20WWdtVWXiQuBB8RocAl0Ga//b+epiGgME5JX
LWXD1aDg/xHy8MUsaMlh6jDfVIFepkPnkwXDpR/n36hpgKa9dErMkgbYeEaPanLH
Yd0kv4xQ36PlMMs9WhoDErGcEG9KxAXN4Axr5wl6PTDn/lXcUFvQoIq/5CSP+Kt5
jq9tK/gRrAc4AWqRugDvQPYUm00Rqzj5Oxm5NVQYDzbyoA66CD68LETuVrfa9GuW
9MAZRO6CDzonAezIdNHsslDb1H8VN/k0zMxjI+0ub4IAmc3I5GfZtvYcpjtMj8L4
2TDS34/COov/Pf2HZ/XXGlzjZ7WPmLl4fdB6hhjs2BsCAwEAAaOCAQYwggECMDAG
CCsGAQUFBwEBBCQwIjAgBggrBgEFBQcwAYYUaHR0cDovL29jc3BzLnNzbC5jb20w
HQYDVR0OBBYEFCYUfuDc16b34tQEJ99h8cLs5zLKMA8GA1UdEwEB/wQFMAMBAf8w
HwYDVR0jBBgwFoAU3QQJB6L1en1SUxKSle44gCUNplkwEQYDVR0gBAowCDAGBgRV
HSAAMDsGA1UdHwQ0MDIwMKAuoCyGKmh0dHA6Ly9jcmxzLnNzbC5jb20vc3NsLmNv
bS1yc2EtUm9vdENBLmNybDAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4ICAQAi6e/iSV5DEqDO6XjQ
SIIzXgc255yv6Oc2sqZnvRyVBHtHvo62jMoHY3Xunc/EofbeS4aHdYBvgkn6CNTj
VkCU+psWwcT3Pg83uP4k4Thu7bXvrClfS+XBlbJiCF/PSJxLrKnxRn+XIGiYl62H
glBhq9K8/fZrI2Qh1mZJmWE0FlxEDCb4i8SBNi8lmDogaFi8/yl32Z9ahmhxcLit
DU/XyKA0yOqvIrOGKH95v+/l8fQkzE1VEFvj+iyv4TXd7mRZDOsfqfIDZhrpou02
kXH/hcXlrR++t8kjj9wt8HHQ+FkryWI6bU3KPRJR6N8EH2EHi23Rp8/kyMs+gwaz
zMqnkNPbMME723rXk6/85sjOUaZCmhmRIx9rgqIWQesU962J0FruGOOasLT7WbZi
FsmSblmpjUAo49sIRi7X493qegyCEAa412ynybhQ7LVsTLEPxVbdmGVih3jVTif/
Nztr2Isaaz4LpMEo4mGCiGxec5mKr1w8AE9n6D91CvxR5/zL1VU1JCVC7sAtkdki
vnN1/6jEKFJvlUr5/FX04JXeomIjXTI8ciruZ6HIkbtJup1n9Zxvmr9JQcFTsP2c
bRbjaT7JD6MBidAWRCJWClR/5etTZwWwWrRCrzvIHC7WO6rCzwu69a+l7ofCKlWs
y702dmPTKEdEfwhgLx0LxJr/Aw==
-----END CERTIFICATE-----

subject=C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com Root Certification Authority RSA

issuer=C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com Root Certification Authority RSA

-----BEGIN CERTIFICATE-----
MIIF3TCCA8WgAwIBAgIIeyyb0xaAMpkwDQYJKoZIhvcNAQELBQAwfDELMAkGA1UE
BhMCVVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQK
DA9TU0wgQ29ycG9yYXRpb24xMTAvBgNVBAMMKFNTTC5jb20gUm9vdCBDZXJ0aWZp
Y2F0aW9uIEF1dGhvcml0eSBSU0EwHhcNMTYwMjEyMTczOTM5WhcNNDEwMjEyMTcz
OTM5WjB8MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hv
dXN0b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjExMC8GA1UEAwwoU1NMLmNv
bSBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IFJTQTCCAiIwDQYJKoZIhvcN
AQEBBQADggIPADCCAgoCggIBAPkP3aMrfcvQKv7sZ4Wm5y4bunfh4/WvpOz6Sl2R
xFdHaxh3a3by/ZPkPQ/CFp4LZsNWlJ4Xg4XOVu/yFv0AYvUiCVToZRdOQbngT0aX
qhvIuG5iXmmxX9sqAn78bMrzQdjt0Oj8P2FI7bADFB0QDksZ4LtO7IZl/zbzXmcC
C52GVWH9ejjt/uIZALdvoVBidXQ8oPrIJZK0bnoix/geoeOy3ZExqysdBP+lSgQ3
6YWkMyv94tZVNHwZpEpox7Ko07fKoZOI68GXvIz5HdkihCR0xwQ9aqkpk8zruFvh
/l8lqjRYyMEjVJ0bmBHDOJx+PYZspQ9AhnwC9FwCTyjLrnGfDzrIM/4RJTXq/LrF
YD3ZfBjVsqnTdXgDciLKOsMf7yzlLqn6niy2UUb9rwPW6mBo6oUWNmuF6R7As93E
JNyAKoFBbZQ+yODJgUEAnl6/f8UImKIYLEJAs/lvOCdLToD0PYFH4Ih86hzOtXVc
US4cK38acijnALXRdMbX5J+tB5O2UzU1/Dfkw/ZdFr4hc96SCvigY2q8lpJqPvi8
ZVWb3vUNiSYE/CUapiVpy8JtynziWV+XrOvvLsi81xtZPCvM8hnIk2snYxnP/Okm
+Mpxm3+T/jRnhE6Z6/yzeAkzcLpmpnbtG3PrGqUNxCITIJRWCk4sbE6x/c+cCbqi
M+2HAgMBAAGjYzBhMB0GA1UdDgQWBBTdBAkHovV6fVJTEpKV7jiAJQ2mWTAPBgNV
HRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFN0ECQei9Xp9UlMSkpXuOIAlDaZZMA4G
A1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAIBgRlCn7Jp0cHh5wYfGV
cpNxJK1ok1iOMq8bs3AD/CUrdIWQPXhq9LmLpZc7tRiRux6n+UBbkflVma8eEdBc
Hadm47GUBwwyOabqG7B52B2ccETjit3E+ZUfijhDPwGFpUenPUayvOUiaPd7nNgs
PgohyC0zrL/FgZkxdMF1ccW+sfAjRfSda/wZY52jvATGGAslu1OJD7OAUN5F7kR/
q5R4ZJjT9ijdh9hwZXT7DrkT66cPYakylszeu+1jTBi7qUD3oFRuIIhxdRjqerQ0
cuAjJ3dctpDqhiVAq+8zD8ufgr6iIPv2tS0a5sKFsXQP+8hlAqRSAUfdSSLBv9jr
a6x+3uxjMxW3IwiPxg+NQVrdjsW5j+VFP3jbutIbQLH+cU0/4IGiul607BXgk90I
H37hVZkLId6Tngr75qNJvTYw/ud3sqB1l7UtgYgXZSD32pAAn8lSzDLKNXz1PQ/Y
K9f1JmzJBjSWFupwWRoyeXkLtoh/D1JIPb9s2KJELtFOt3JY04kTlf5Eq/jXixtu
nLwsoFvVagCvXzfh1foQC5ichucmj87w7G6KVwuA406ywKBjYZC6VWg3dGq2ktuf
oYYitmUnDuy2n0Jg5GfCtdpBC8TTi2EbvPofkSvXRAdeuims2cXp71NIWuuA8ShY
Ic2wBlX7Jz9TkHCpBB5XJ7k=
-----END CERTIFICATE-----

Thanks for looking into this!

Answer 3 · 2024-09-10T13:09:12.000Z

Thanks! I'm getting ready for a presentation next week but I'll try to revisit this soon

Answer 4 · 2024-09-25T20:00:57.000Z

Wait, does the file literally contain lines like subject=C = US, ST = Texas, L = Houston, O = SSL Corporation, CN = SSL.com Root Certification Authority RSA ?

If so, that's the problem. PEM files can only contain PEM blocks.

Answer 5 · 2024-09-30T08:44:47.000Z

@mholt I believe the tls package handles them because it was working with these as well.

FWIW, the certificates are stored without the comments -- so only PEM blocks.

You can check an example here. The certificate chain seems to missing but if you visit the page through your browser you should see that the certificate is loading. But if you use a tool like curl, the handshake will fail because the certificate is unknown.

Answer 6 · 2024-09-30T10:08:56.000Z

Ok, thanks. Next question sounds stupid, but can you print and paste here the exact output of appConfig.CertValue from the code above? log.Println(string(appConfig.CertValue)) in the line immediately before the Cache call should suffice.

Answer 7 · 2024-09-30T10:24:01.000Z

Ok, thanks. Next question sounds stupid, but can you print and paste here the exact output of appConfig.CertValue from the code above? log.Println(string(appConfig.CertValue)) in the line immediately before the Cache call should suffice.

@mholt is there an alternative way to debug this? They're self-hosting Stormkit so I'd need to publish a version just with the debug 😅

Answer 8 · 2024-09-30T15:45:52.000Z

Not without the exact code and inputs to it. Taking your PEM blocks above (without the extra lines as mentioned) and giving that as input to cnf.CacheUnmanagedCertificatePEMBytes() works perfectly for me. So I'd need a full reproducer to verify there is a bug.

Answer 9 · 2024-10-02T12:34:53.000Z

@mholt this is the output as you requested - right before calling cacheUnmanagedCertificatePEMBytes:

Answer 10 · 2024-10-04T20:48:09.000Z

Ah, I guess to test this and reproduce the issue, I will need both a public key (certificate) and private key pair -- i.e. the private key associated with the cert.

Can you post a test cert chain and private key that I can use to reproduce the issue? Don't send me your production private key, obviously. :)

Answer 11 · 2024-10-07T13:42:44.000Z

@mholt would https://www.ssl.com/certificates/free/buy/ work for you to generate a test certificate? This is where I'd generate the test certificate anyways. I think it's the easiest way and prevents back and forth.

Answer 12 · 2024-10-07T14:27:55.000Z

Well that's the thing, I can generate my own cert chain and everything works as expected.

That's why I want to test with your specific cert chain you're having an issue with.

Answer 13 · 2024-11-18T19:28:04.000Z

@mholt I still have this in mind. Apologies that I couldn't reply lately, been busy with other things. I'll ping you once I have a test certificate set.