aws/s2n-tls

6 tests failing with Error Message: 'Certificate is untrusted' when compiling on RHEL 9

wombelix opened this issue · 8 comments

Problem:

I try to compile s2n-tls on different Fedora and Red Hat Enterprise Linux versions with shared libraries against OpenSSL.

On RHEL 9, six tests failing with the Error Message: Certificate is untrusted
All tests passing on RHEL 8, Fedora 39, Fedora 40 and Fedora Rawhide (x86_64, aarch64).
The result / error is identical when I drop build option -DBUILD_SHARED_LIBS=ON.

I'm looking for guidance how to further troubleshoot and solve the issue.

Steps to reproduce on a fresh RHEL 9.3 instance with AMI RHEL-9.3.0_HVM-20240117-x86_64-49-Hourly2-GP3:

Additional packages installed: cmake, gcc, git, openssl-devel

Versions: cmake-3.26.5, gcc-11.4.1, openssl-3.0.7

Build steps:

git clone https://github.com/aws/s2n-tls.git
cd s2n-tls

cmake . -Bbuild \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_INSTALL_PREFIX=./s2n-tls-install
cmake --build build -j $(nproc)
S2N_PRINT_STACKTRACE=1 CTEST_PARALLEL_LEVEL=$(nproc) ctest --test-dir build --output-on-failure
cmake --install build

Relevant logs of failing tests with stacktrace output enabled.

[...]

Internal ctest changing into directory: /home/ec2-user/s2n-tls/build
Test project /home/ec2-user/s2n-tls/build
        Start  17: s2n_cert_validation_callback_test
        Start  55: s2n_config_test
        Start  63: s2n_crl_test
        Start 130: s2n_mutual_auth_test
  1/263 Test  #17: s2n_cert_validation_callback_test ................***Failed    0.11 sec
Running /home/ec2-user/s2n-tls/tests/unit/s2n_cert_validation_callback_test.c ... 
Stacktrace is:
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_calculate_stacktrace+0x5c) [0x7fe63f1e171c]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_x509_validator_validate_cert_chain+0x1346) [0x7fe63f2313c6]
/home/ec2-user/s2n-tls/build/bin/s2n_cert_validation_callback_test(main+0x555) [0x403b75]
/lib64/libc.so.6(+0x3feb0) [0x7fe63e83feb0]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7fe63e83ff60]
/home/ec2-user/s2n-tls/build/bin/s2n_cert_validation_callback_test(_start+0x25) [0x40ec35]
FAILED test 152
s2n_result_is_ok(s2n_x509_validator_validate_cert_chain(&validator, conn, chain_data, chain_len, &pkey_type, &public_key_out)) is not true  (/home/ec2-user/s2n-tls/tests/unit/s2n_cert_validation_callback_test.c:239)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /home/ec2-user/s2n-tls/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

        Start 262: s2n_x509_validator_test
  2/263 Test #262: s2n_x509_validator_test ..........................***Failed    0.07 sec
Running /home/ec2-user/s2n-tls/tests/unit/s2n_x509_validator_test.c ... 
Stacktrace is:
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_calculate_stacktrace+0x5c) [0x7f8f397a671c]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_x509_validator_validate_cert_chain+0x1346) [0x7f8f397f63c6]
/home/ec2-user/s2n-tls/build/bin/s2n_x509_validator_test(main+0x9c0) [0x403f40]
/lib64/libc.so.6(+0x3feb0) [0x7f8f38e3feb0]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f8f38e3ff60]
/home/ec2-user/s2n-tls/build/bin/s2n_x509_validator_test(_start+0x25) [0x433705]
FAILED test 50
s2n_result_is_ok(s2n_x509_validator_validate_cert_chain(&validator, connection, chain_data, chain_len, &pkey_type, &public_key_out)) is not true  (/home/ec2-user/s2n-tls/tests/unit/s2n_x509_validator_test.c:334)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /home/ec2-user/s2n-tls/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

        Start 263: s2n_x509_validator_time_verification_test
  3/263 Test  #63: s2n_crl_test .....................................***Failed    0.74 sec
Running /home/ec2-user/s2n-tls/tests/unit/s2n_crl_test.c   ... 
Stacktrace is:
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_calculate_stacktrace+0x5c) [0x7fced72a471c]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_x509_validator_validate_cert_chain+0x1346) [0x7fced72f43c6]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_server_cert_recv+0xaa) [0x7fced72e7baa]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(+0x79104) [0x7fced72cd104]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_negotiate_impl+0x158) [0x7fced72ce5d8]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_negotiate+0x28) [0x7fced72ced28]
/home/ec2-user/s2n-tls/build/bin/s2n_crl_test(s2n_negotiate_test_server_and_client+0xe8) [0x41fde8]
/home/ec2-user/s2n-tls/build/bin/s2n_crl_test(main+0x308d) [0x4066cd]
/lib64/libc.so.6(+0x3feb0) [0x7fced6a3feb0]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7fced6a3ff60]
/home/ec2-user/s2n-tls/build/bin/s2n_crl_test(_start+0x25) [0x41e685]
FAILED test 341
!(((s2n_negotiate_test_server_and_client(server_conn, client_conn))) == (-1)) is not true  (/home/ec2-user/s2n-tls/tests/unit/s2n_crl_test.c:782)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /home/ec2-user/s2n-tls/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

        Start   2: s2n_aead_aes_test
  4/263 Test  #55: s2n_config_test ..................................***Failed   25.76 sec
Running /home/ec2-user/s2n-tls/tests/unit/s2n_config_test.c ... 
Stacktrace is:
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_calculate_stacktrace+0x5c) [0x7fb5173aa71c]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_x509_validator_validate_cert_chain+0x1346) [0x7fb5173fa3c6]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_server_cert_recv+0xaa) [0x7fb5173edbaa]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(+0x79104) [0x7fb5173d3104]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_negotiate_impl+0x158) [0x7fb5173d45d8]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_negotiate+0x28) [0x7fb5173d4d28]
/home/ec2-user/s2n-tls/build/bin/s2n_config_test(s2n_negotiate_test_server_and_client+0xe8) [0x42a068]
/home/ec2-user/s2n-tls/build/bin/s2n_config_test(main+0x2e13) [0x407643]
/lib64/libc.so.6(+0x3feb0) [0x7fb516a3feb0]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7fb516a3ff60]
/home/ec2-user/s2n-tls/build/bin/s2n_config_test(_start+0x25) [0x428655]
FAILED test 855
!(((s2n_negotiate_test_server_and_client(server_conn, client_conn))) == (-1)) is not true  (/home/ec2-user/s2n-tls/tests/unit/s2n_config_test.c:948)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /home/ec2-user/s2n-tls/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

        Start 153: s2n_random_test
  5/263 Test #263: s2n_x509_validator_time_verification_test ........***Failed   25.79 sec
Running /home/ec2-user/s2n-tls/tests/unit/s2n_x509_validator_time_verification_test.c ... 
Stacktrace is:
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_calculate_stacktrace+0x5c) [0x7f164a54371c]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_x509_validator_validate_cert_chain+0x1346) [0x7f164a5933c6]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_server_cert_recv+0xaa) [0x7f164a586baa]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(+0x79104) [0x7f164a56c104]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_negotiate_impl+0x158) [0x7f164a56d5d8]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_negotiate+0x28) [0x7f164a56dd28]
/home/ec2-user/s2n-tls/build/bin/s2n_x509_validator_time_verification_test(s2n_negotiate_test_server_and_client+0xe8) [0x40bbf8]
/home/ec2-user/s2n-tls/build/bin/s2n_x509_validator_time_verification_test(main+0x961) [0x403f11]
/lib64/libc.so.6(+0x3feb0) [0x7f1649c3feb0]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f1649c3ff60]
/home/ec2-user/s2n-tls/build/bin/s2n_x509_validator_time_verification_test(_start+0x25) [0x40a5e5]
FAILED test 144
(s2n_errno) == (expected_error) is not true  (/home/ec2-user/s2n-tls/tests/unit/s2n_x509_validator_time_verification_test.c:230)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /home/ec2-user/s2n-tls/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

        Start 184: s2n_self_talk_nonblocking_test
  6/263 Test #130: s2n_mutual_auth_test .............................***Failed   28.78 sec
Running /home/ec2-user/s2n-tls/tests/unit/s2n_mutual_auth_test.c ... 
Stacktrace is:
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_calculate_stacktrace+0x5c) [0x7f6ca93fa71c]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_x509_validator_validate_cert_chain+0x1346) [0x7f6ca944a3c6]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_server_cert_recv+0xaa) [0x7f6ca943dbaa]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(+0x79104) [0x7f6ca9423104]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_negotiate_impl+0x158) [0x7f6ca94245d8]
/home/ec2-user/s2n-tls/build/lib/libs2n.so.1(s2n_negotiate+0x28) [0x7f6ca9424d28]
/home/ec2-user/s2n-tls/build/bin/s2n_mutual_auth_test(s2n_negotiate_test_server_and_client+0xe8) [0x40cf78]
/home/ec2-user/s2n-tls/build/bin/s2n_mutual_auth_test(main+0x471) [0x4039a1]
/lib64/libc.so.6(+0x3feb0) [0x7f6ca8a3feb0]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f6ca8a3ff60]
/home/ec2-user/s2n-tls/build/bin/s2n_mutual_auth_test(_start+0x25) [0x40b965]
FAILED test 28
!(((s2n_negotiate_test_server_and_client(server_conn, client_conn))) == (-1)) is not true  (/home/ec2-user/s2n-tls/tests/unit/s2n_mutual_auth_test.c:123)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /home/ec2-user/s2n-tls/tls/s2n_x509_validator.c:721'
 System Error: Input/output error (5)

[...]

98% tests passed, 6 tests failed out of 263

Label Time Summary:
unit    = 832.76 sec*proc (263 tests)

Total Test time (real) = 267.61 sec

The following tests FAILED:
	 17 - s2n_cert_validation_callback_test (Failed)
	 55 - s2n_config_test (Failed)
	 63 - s2n_crl_test (Failed)
	130 - s2n_mutual_auth_test (Failed)
	262 - s2n_x509_validator_test (Failed)
	263 - s2n_x509_validator_time_verification_test (Failed)
Errors while running CTest

Thanks for the issue! That is a very interesting set of failures. I don't think we test on any RHEL in our CI 🤔 not sure how we would add that.

While not the exact AMI you're using, we've had success with unit tests on AL2023, which is similar to FC34/35, with Openssl 3.0.x. I was unable to repro this error on FC35, using the steps provided.

One thing that might help is changing your build target to -DCMAKE_BUILD_TYPE=RelwithDebInfo or even just Debug. Another thing to consider is that the error might not be 100% accurate as it's the default in this case statement.

While not the exact AMI you're using, we've had success with unit tests on AL2023, which is similar to FC34/35, with Openssl 3.0.x. I was unable to repro this error on FC35, using the steps provided.

The issue only occurs on RHEL 9, so I'm not exactly sure why you refer to F34/F35 in this context? Because F34 was the base for CentOS 9 Stream which is the base for RHEL 9?

I'm no expert in the s2n-tls code base but also saw the case statement you mentioned (https://github.com/aws/s2n-tls/blob/main/tls/s2n_x509_validator.c#L721). My first thought was, if the tests run against the system wide trusted certs, maybe the tests certs are signed by ca that's not trusted anymore or use a hash that's unsupported in current RHEL 9 releases. Or because of compliance requirements openssl is compiled with different flags, therefore more strict as on other OS versions and fails.

I can test the changed build flag and report the results back.
Any other suggestions how it can be troubleshooted? The default case statement hides it away a bit.

I did a build with -DCMAKE_BUILD_TYPE=RelwithDebInfo rhel9_relwithdebinfo_builder-live.log.gz and -DCMAKE_BUILD_TYPE=Debug rhel9_debug_builder-live.log.gz on RHEL9. Both times the same tests failed again with the untrusted certification issues.

I had another idea but it's also not the problem or at least not the part of s2n-tls I thought is responsible.
But let me share it so someone with more s2n-tls knowledge can take a look too:

Reflecting on the fact that it is a problem that right now only occurs on RHEL9, the feature flag S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH was drawing my attention:

+ /usr/bin/cmake -S . -B redhat-linux-build -DCMAKE_C_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_CXX_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_Fortran_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_INSTALL_DO_STRIP:BOOL=OFF -DCMAKE_INSTALL_PREFIX:PATH=/usr -DINCLUDE_INSTALL_DIR:PATH=/usr/include -DLIB_INSTALL_DIR:PATH=/usr/lib64 -DSYSCONF_INSTALL_DIR:PATH=/etc -DSHARE_INSTALL_PREFIX:PATH=/usr/share -DLIB_SUFFIX=64 -DBUILD_SHARED_LIBS:BOOL=ON -GNinja -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=RelwithDebInfo
-- The C compiler identification is GNU 11.4.1
[...]

-- feature S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH: TRUE

[...]

Red Hat has set SHA1 on path to deprecation and started with it in RHEL 9:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/VVLHQAWI3IQ7NRLKMUHJ27JV3V2JAFDP/
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/considerations_in_adopting_rhel_9/assembly_security_considerations-in-adopting-rhel-9#ref_considerations-security-crypto_changes-to-security

In Fedora the patch to disable SHA1 in the openssl package still allows SHA1 by default:
https://src.fedoraproject.org/rpms/openssl/blob/rawhide/f/0049-Allow-disabling-of-SHA1-signatures.patch#_131

But in CentOS 9 Stream and therefore RHEL 9 the patch is changed to disable SHA1 by default:
https://gitlab.com/redhat/centos-stream/rpms/openssl/-/blob/c9s/0049-Selectively-disallow-SHA1-signatures.patch?ref_type=heads#L131

This patch leaves EVP_md5_sha1 in openssl/evp.h of the openssl-devel package untouched. My understanding is that therefore https://github.com/aws/s2n-tls/blob/main/tests/features/S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.c in combination with #3921 and #4074 makes the false assumption that openSSL provides support for SHA1 and enables the feature.

To test my theory I removed the feature by applying this patch:

From 046f65355835d0a186efc71a15b892fd2730f773 Mon Sep 17 00:00:00 2001
From: Dominik Wombacher <dominik@wombacher.cc>
Date: Fri, 17 May 2024 09:07:54 +0000
Subject: [PATCH] fix: Disable feature
 'S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH'

---
 ...S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.c | 20 -------------------
 ...LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.flags |  0
 2 files changed, 20 deletions(-)
 delete mode 100644 tests/features/S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.c
 delete mode 100644 tests/features/S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.flags

diff --git a/tests/features/S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.c b/tests/features/S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.c
deleted file mode 100644
index baa928e1..00000000
--- a/tests/features/S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.c
+++ /dev/null
@@ -1,20 +0,0 @@
-/*
- * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
- *
- * Licensed under the Apache License, Version 2.0 (the "License").
- * You may not use this file except in compliance with the License.
- * A copy of the License is located at
- *
- *  http://aws.amazon.com/apache2.0
- *
- * or in the "license" file accompanying this file. This file is distributed
- * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
- * express or implied. See the License for the specific language governing
- * permissions and limitations under the License.
- */
-
-#include <openssl/evp.h>
-int main() {
-    EVP_md5_sha1();
-    return 0;
-}
diff --git a/tests/features/S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.flags b/tests/features/S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH.flags
deleted file mode 100644
index e69de29b..00000000
-- 
2.45.0

But this didn't change anything, the same tests failing and claiming the cert is untrusted:

The following tests FAILED:
	 17 - s2n_cert_validation_callback_test (Failed)
	 55 - s2n_config_test (Failed)
	 63 - s2n_crl_test (Failed)
	130 - s2n_mutual_auth_test (Failed)
	262 - s2n_x509_validator_test (Failed)
	263 - s2n_x509_validator_time_verification_test (Failed)
Errors while running CTest

So maybe it is related to the SHA1 deprecation and I was looking at the wrong part of the code.
Or the failed tests have a different root cause and not related to the SHA1 deprecation that was started in RHEL 9.
I can't tell for sure, too little C programming skills.

Using the build type RelwithDebInfo will build s2n-tls with debug symbols. This allows you to use gdb with s2n-tls and pause the tests in a specific location. In particular, it would be interesting to see what the actual value of ossl_error is here in one of the failing tests, because it's not matching one of our expected cases.

The note about SHA1 in RHEL 9 is intriguing. However, the feature flag you mentioned isn't actually used to turn on/off SHA1 support, which is why removing it does nothing.

Your guess about the issue being a SHA1 signature algorithm is a good one. The failing tests use a certificate which uses sha1WithRSAEncryption as its signature algorithm.

This diff changes the s2n_config_test to use a certificate with a sha256 signature algorithm instead of one with the sha1 signature algorithm. If it passes on RHEL 9, then that further supports the theory that its a sha1 problem.

diff --git a/tests/unit/s2n_config_test.c b/tests/unit/s2n_config_test.c
index d4fe8d7d9..05f461dd2 100644
--- a/tests/unit/s2n_config_test.c
+++ b/tests/unit/s2n_config_test.c
@@ -910,7 +910,7 @@ int main(int argc, char **argv)
             DEFER_CLEANUP(struct s2n_cert_chain_and_key *chain_and_key = NULL,
                     s2n_cert_chain_and_key_ptr_free);
             EXPECT_SUCCESS(s2n_test_cert_chain_and_key_new(&chain_and_key,
-                    S2N_DEFAULT_TEST_CERT_CHAIN, S2N_DEFAULT_TEST_PRIVATE_KEY));
+                    S2N_RSA_2048_PKCS8_CERT_CHAIN, S2N_RSA_2048_PKCS8_KEY));
 
             /* Ensure a handshake succeeds with a minimal server config and no mutual auth */
             {
@@ -927,13 +927,13 @@ int main(int argc, char **argv)
                 DEFER_CLEANUP(struct s2n_config *client_config = s2n_config_new_minimal(), s2n_config_ptr_free);
                 EXPECT_NOT_NULL(client_config);
                 EXPECT_SUCCESS(s2n_config_set_cipher_preferences(client_config, "default"));
-                EXPECT_SUCCESS(s2n_config_set_verification_ca_location(client_config, S2N_DEFAULT_TEST_CERT_CHAIN, NULL));
+                EXPECT_SUCCESS(s2n_config_set_verification_ca_location(client_config, S2N_RSA_2048_PKCS8_CERT_CHAIN, NULL));
 
                 DEFER_CLEANUP(struct s2n_connection *client_conn = s2n_connection_new(S2N_CLIENT),
                         s2n_connection_ptr_free);
                 EXPECT_NOT_NULL(client_conn);
                 EXPECT_SUCCESS(s2n_connection_set_config(client_conn, client_config));
-                EXPECT_SUCCESS(s2n_set_server_name(client_conn, "s2nTestServer"));
+                EXPECT_SUCCESS(s2n_set_server_name(client_conn, "FakeRoot"));
 
                 struct s2n_test_io_pair io_pair = { 0 };
                 EXPECT_SUCCESS(s2n_io_pair_init_non_blocking(&io_pair));

Your guess about the issue being a SHA1 signature algorithm is a good one. The failing tests use a certificate which uses sha1WithRSAEncryption as its signature algorithm.

This diff changes the s2n_config_test to use a certificate with a sha256 signature algorithm instead of one with the sha1 signature algorithm. If it passes on RHEL 9, then that further supports the theory that its a sha1 problem.

Thanks @maddeleine, I applied the patch and did a new build.
Now we have five instead of six tests failing, which seems like the expected result because 55 - s2n_config_test passed this time.

So I would say we getting closer and it seems indeed relation to the SHA1 deprecation.

Still failing:

	 17 - s2n_cert_validation_callback_test (Failed)
	 63 - s2n_crl_test (Failed)
	130 - s2n_mutual_auth_test (Failed)
	262 - s2n_x509_validator_test (Failed)
	263 - s2n_x509_validator_time_verification_test (Failed)

Details

 16/263 Test  #17: s2n_cert_validation_callback_test ................***Failed    0.06 sec
Running /builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_cert_validation_callback_test.c ... NOTE: Some details are omitted, run with S2N_PRINT_STACKTRACE=1 for a verbose backtrace.
See https://github.com/aws/s2n-tls/blob/main/docs/usage-guide
FAILED test 152
s2n_result_is_ok(s2n_x509_validator_validate_cert_chain(&validator, conn, chain_data, chain_len, &pkey_type, &public_key_out)) is not true  (/builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_cert_validation_callback_test.c:239)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /builddir/build/BUILD/s2n-tls-1.4.14/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

 62/263 Test  #63: s2n_crl_test .....................................***Failed    0.55 sec
Running /builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_crl_test.c ... NOTE: Some details are omitted, run with S2N_PRINT_STACKTRACE=1 for a verbose backtrace.
See https://github.com/aws/s2n-tls/blob/main/docs/usage-guide
FAILED test 341
!(((s2n_negotiate_test_server_and_client(server_conn, client_conn))) == (-1)) is not true  (/builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_crl_test.c:782)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /builddir/build/BUILD/s2n-tls-1.4.14/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

152/263 Test #130: s2n_mutual_auth_test .............................***Failed   24.54 sec
Running /builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_mutual_auth_test.c ... NOTE: Some details are omitted, run with S2N_PRINT_STACKTRACE=1 for a verbose backtrace.
See https://github.com/aws/s2n-tls/blob/main/docs/usage-guide
FAILED test 28
!(((s2n_negotiate_test_server_and_client(server_conn, client_conn))) == (-1)) is not true  (/builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_mutual_auth_test.c:123)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /builddir/build/BUILD/s2n-tls-1.4.14/tls/s2n_x509_validator.c:721'
 System Error: Input/output error (5)

262/263 Test #262: s2n_x509_validator_test ..........................***Failed    0.04 sec
Running /builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_x509_validator_test.c ... NOTE: Some details are omitted, run with S2N_PRINT_STACKTRACE=1 for a verbose backtrace.
See https://github.com/aws/s2n-tls/blob/main/docs/usage-guide
FAILED test 50
s2n_result_is_ok(s2n_x509_validator_validate_cert_chain(&validator, connection, chain_data, chain_len, &pkey_type, &public_key_out)) is not true  (/builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_x509_validator_test.c:334)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /builddir/build/BUILD/s2n-tls-1.4.14/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

263/263 Test #263: s2n_x509_validator_time_verification_test ........***Failed   26.26 sec
Running /builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_x509_validator_time_verification_test.c ... NOTE: Some details are omitted, run with S2N_PRINT_STACKTRACE=1 for a verbose backtrace.
See https://github.com/aws/s2n-tls/blob/main/docs/usage-guide
FAILED test 144
(s2n_errno) == (expected_error) is not true  (/builddir/build/BUILD/s2n-tls-1.4.14/tests/unit/s2n_x509_validator_time_verification_test.c:230)
Error Message: 'Certificate is untrusted'
 Debug String: 'Error encountered in /builddir/build/BUILD/s2n-tls-1.4.14/tls/s2n_x509_validator.c:721'
 System Error: Success (0)

To resolve this it might make sense for us to just update the default certificate chain to one that doesn't use SHA1.