dotnet/runtime

HTTPS requests fail on Ubuntu 24.04 Noble ARM32 due to bundled certs "NotTimeValid" error

lbussell opened this issue · 44 comments

lbussell commented

Description

HTTPS requests from .NET are failing on Ubuntu 24.04 Noble on ARM32 due to bundled certs "NotTimeValid" errors.

I believe this may be because 24.04 has migrated to 64-bit time. The OpenSSL package has changed from libssl3 to libssl3t64.

Reproduction Steps

  1. dotnet new console
  2. dotnet add package System.Text.Json
  3. .NET will try to restore the package from NuGet.org and fail with the error under "Actual behavior" below.

I created this repro Dockerfile. It may or may not work since I had issues with it on my AMD64 dev machine. No easy way for me to test this currently outside of our official .NET Container image builds. With some trial and error you can probably get the issue to repro on a real arm32 machine using Docker. https://gist.github.com/lbussell/52e0ac904108d238d0e511f8b6ec89e1

Expected behavior

The .NET CLI should hit the NuGet endpoint and complain that the package doesn't exist.

Actual behavior

From our PR validation in .NET Docker: https://dev.azure.com/dnceng-public/public/_build/results?buildId=652911&view=logs&j=7bc65791-3246-5ca2-874f-59d2e579cf6b&t=08651ed6-ba3f-5f8a-52df-50083ac157c2&l=942

 #10 2.407 info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/share/dotnet/sdk/8.0.204/trustedroots/codesignctl.pem'.
 #10 2.491 info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/share/dotnet/sdk/8.0.204/trustedroots/timestampctl.pem'.
 #10 2.494 info : Adding PackageReference for package 'foo' into project '/source/app/app.csproj'.
 #10 8.602 error: Unable to load the service index for source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json.
 #10 8.602 error:   The SSL connection could not be established, see inner exception.
 #10 8.602 error:   The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

Regression?

No response

Known Workarounds

No response

Configuration

  • .NET Runtime: 8.0.4 and 9.0.0-preview.4.24218.7
  • OS: Ubuntu 24.04 "Noble"
  • Arch: ARM
  • Do you know whether it is specific to that configuration? Yes it is specific to this config. Likely affects older .NET versions as well.

Other information

Discovered in dotnet/dotnet-docker#5241

dotnet-policy-service commented

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

vcsjones commented

I can take a look (and finally have an excuse to take that Raspberry Pi out of the box!)

vcsjones commented

Firstly

  1. I can reproduce it

    Distributor ID:	Ubuntu
Description:	Ubuntu Noble Numbat (development branch)
Release:	24.04
Codename:	noble
root@1dc47c31f061:/project# dotnet add package System.Text.Json
  Determining projects to restore...
  Writing /tmp/tmpiHt9zR.tmp
info : X.509 certificate chain validation will use the fallback certificate bundle at '/root/dotnet/sdk/8.0.204/trustedroots/codesignctl.pem'.
info : X.509 certificate chain validation will use the fallback certificate bundle at '/root/dotnet/sdk/8.0.204/trustedroots/timestampctl.pem'.
info : Adding PackageReference for package 'System.Text.Json' into project '/project/project.csproj'.
error: Unable to load the service index for source https://api.nuget.org/v3/index.json.
error:   The SSL connection could not be established, see inner exception.
error:   The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

  2. .NET "knows" it is the right time. So it's not like the clock is wrong, or that the source of time is wrong.

    root@1dc47c31f061:/project# date
Tue Apr 23 21:40:27 EDT 2024
root@1dc47c31f061:/project# dotnet run
Hello, World!04/23/2024 21:40:37
root@1dc47c31f061:/project#

  3. OpenSSL CLI is okay with the host. So, it's not like the certificate really is expired or there is some underlying problem in OpenSSL.

    No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 4273 bytes and written 441 bytes
Verification: OK
---

  4. The X509Certificate2 is able to load the notBefore and notAfter correctly. (.NET and OpenSSL differ by my timezone offset since .NET is showing it in local time and OpenSSL is showing it in UTC time)

    root@d2b82c600199:/project# dotnet run
10/18/2023 16:00:08
10/12/2024 16:00:08
Hello, World!
root@d2b82c600199:/project# openssl x509 -in cert.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            33:00:0d:53:71:cb:65:3d:f3:ee:7c:dc:5c:00:00:00:0d:53:71
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: C = US, O = Microsoft Corporation, CN = Microsoft Azure RSA TLS Issuing CA 07
        Validity
            Not Before: Oct 18 20:00:08 2023 GMT
            Not After : Oct 12 20:00:08 2024 GMT

This will take a bit of additional time to troubleshoot, whatever is going on is not immediately obvious (to me)