OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

Question

OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

jamesbrooks94 opened this issue 3 years ago · 96 comments

Hey,

For the past hour or so I’ve been getting this error while using this action:
Error: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

I’ve checked the thumbprint and it’s correctly set, would this be an issue with GitHub, are there any known issues at the moment?

Answer 1 · 2022-01-13T00:31:43.000Z

Additionally (this may be completely unrelated, if so I will open a new issue) intermittently we are getting this error from the action

Run aws-actions/configure-aws-credentials@ea7b857d8a33dc2fb4ef5a724500044281b49a5e
with:
    role-to-assume: arn:aws:iam::***:role/***
    role-duration-seconds: 3600
    aws-region: eu-west-1

Error: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers

Answer 2 · 2022-01-13T00:32:33.000Z

I found a tweet.

https://twitter.com/arkadiyt/status/1481418230082248714

For anyone using Github Actions -> AWS role assumption & suddenly getting "OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint" - Github updated their certificate chain and the new thumbprint to use is 6938fd4d98bab03faadb97b34396831e3780aea1

Answer 3 · 2022-01-13T00:39:06.000Z

Thanks @suzuki-shunsuke I've also noticed there’s a PR open already, however the thumbprints are different!

https://github.com/aws-actions/configure-aws-credentials/pull/355/files

Answer 4 · 2022-01-13T01:33:52.000Z

I could obtain the new thumbprint as the AWS official doc https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html.

% curl https://token.actions.githubusercontent.com/.well-known/openid-configuration
{"issuer":"https://token.actions.githubusercontent.com","jwks_uri":"https://token.actions.githubusercontent.com/.well-known/jwks","subject_types_supported":["public","pairwise"],"response_types_supported":["id_token"],"claims_supported":["sub","aud","exp","iat","iss","jti","nbf","ref","repository","repository_owner","run_id","run_number","run_attempt","actor","workflow","head_ref","base_ref","event_name","ref_type","environment","job_workflow_ref"],"id_token_signing_alg_values_supported":["RS256"],"scopes_supported":["openid"]}%

% openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443
CONNECTED(00000005)
...
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
-----BEGIN CERTIFICATE-----
MIIE6jCCA9KgAwIBAgIQCjUI1VwpKwF9+K1lwA/35DANBgkqhkiG9w0BAQsFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
QTAeFw0yMDA5MjQwMDAwMDBaFw0zMDA5MjMyMzU5NTlaME8xCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxKTAnBgNVBAMTIERpZ2lDZXJ0IFRMUyBS
U0EgU0hBMjU2IDIwMjAgQ0ExMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAwUuzZUdwvN1PWNvsnO3DZuUfMRNUrUpmRh8sCuxkB+Uu3Ny5CiDt3+PE0J6a
qXodgojlEVbbHp9YwlHnLDQNLtKS4VbL8Xlfs7uHyiUDe5pSQWYQYE9XE0nw6Ddn
g9/n00tnTCJRpt8OmRDtV1F0JuJ9x8piLhMbfyOIJVNvwTRYAIuE//i+p1hJInuW
raKImxW8oHzf6VGo1bDtN+I2tIJLYrVJmuzHZ9bjPvXj1hJeRPG/cUJ9WIQDgLGB
Afr5yjK7tI4nhyfFK3TUqNaX3sNk+crOU6JWvHgXjkkDKa77SU+kFbnO8lwZV21r
eacroicgE7XQPUDTITAHk+qZ9QIDAQABo4IBrjCCAaowHQYDVR0OBBYEFLdrouqo
qoSMeeq02g+YssWVdrn0MB8GA1UdIwQYMBaAFAPeUDVW0Uy7ZvCj4hsbw5eyPdFV
MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
EgYDVR0TAQH/BAgwBgEB/wIBADB2BggrBgEFBQcBAQRqMGgwJAYIKwYBBQUHMAGG
GGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBABggrBgEFBQcwAoY0aHR0cDovL2Nh
Y2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0R2xvYmFsUm9vdENBLmNydDB7BgNV
HR8EdDByMDegNaAzhjFodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vRGlnaUNlcnRH
bG9iYWxSb290Q0EuY3JsMDegNaAzhjFodHRwOi8vY3JsNC5kaWdpY2VydC5jb20v
RGlnaUNlcnRHbG9iYWxSb290Q0EuY3JsMDAGA1UdIAQpMCcwBwYFZ4EMAQEwCAYG
Z4EMAQIBMAgGBmeBDAECAjAIBgZngQwBAgMwDQYJKoZIhvcNAQELBQADggEBAHer
t3onPa679n/gWlbJhKrKW3EX3SJH/E6f7tDBpATho+vFScH90cnfjK+URSxGKqNj
OSD5nkoklEHIqdninFQFBstcHL4AGw+oWv8Zu2XHFq8hVt1hBcnpj5h232sb0HIM
ULkwKXq/YFkQZhM6LawVEWwtIwwCPgU7/uWhnOKK24fXSuhe50gG66sSmvKvhMNb
g0qZgYOrAKHKCjxMoiWJKiKnpPMzTFuMLhoClw+dj20tlQj7T9rxkTgl4ZxuYRiH
as6xuwAwapu3r9rxxZf+ingkquqTgLozZXq8oXfpf2kUCwA/d5KxTVtzhwoT0JzI
8ks5T1KESaZMkE4f97Q=
-----END CERTIFICATE-----
...

# saved the root certificate as cert.crt

% openssl x509 -in cert.crt -fingerprint -noout | sed -e 's/://g'
SHA1 Fingerprint=6938FD4D98BAB03FAADB97B34396831E3780AEA1

% openssl x509 -in Downloads/cert.crt -fingerprint -noout | sed -e 's/://g' | tr '[:upper:]' '[:lower:]'
sha1 fingerprint=6938fd4d98bab03faadb97b34396831e3780aea1

It seems the new thumbprint is 6938fd4d98bab03faadb97b34396831e3780aea1.

Answer 5 · 2022-01-13T07:38:10.000Z

FYI you can also run this command:

openssl s_client -connect token.actions.githubusercontent.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
SHA1 Fingerprint=15:E2:91:08:71:81:11:E5:9B:3D:AD:31:95:46:47:E3:C3:44:A2:31

Just remove the : and lowercase everything

Answer 6 · 2022-01-13T09:19:13.000Z

How to get new thumbprint in advance? This will break once every year.

Answer 7 · 2022-01-13T09:23:44.000Z

@jakejscott that command is giving a different fingerprint because you take the first cert of the chain, not the last as doc suggests.

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html

If you see more than one certificate, find the last certificate displayed (at the end of the command output). This contains the certificate of the top intermediate CA in the certificate authority chain.

Answer 8 · 2022-01-13T12:09:17.000Z

Does anyone know if this was this replaced due to expiration of the old cert, or a security issue? I'd like to know whether to add the new cert in addition to the old one, or should I replace the old thumbprint.

Answer 9 · 2022-01-13T12:52:42.000Z

For those using terraform to manage the OIDC provider in AWS:

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
  client_id_list  = ["sts.amazonaws.com"]
}

Answer 10 · 2022-01-13T15:02:16.000Z

I understand there's a new thumbprint, but how exactly do we apply this to the github action?

      - name: Configure AWS credentials via role assumption
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::<my_account_id>:role/<my_role>
          role-duration-seconds: 1800 # the ttl of the session, in seconds.
          aws-region: us-east-1

Can we provide the new thumbprint here, or what exactly must be done? Does this get applied on the AWS side? A bit confused. but still researching.

Answer 11 · 2022-01-13T15:18:43.000Z

@altjx has to be updated in AWS IAM --> Identity Providers -->(token.actions.githubusercontent.com)
Find Thumbprints on right side and add new one 6938fd4d98bab03faadb97b34396831e3780aea1
or through cloudformation/terraform is you are using IaC. But, also, I am expecting some

Answer 12 · 2022-01-13T16:19:06.000Z

FYI, even after 12h+ since the change, I'm still getting two different cert chains from the same hostname. Simple test:

while true; do echo | openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443 2>&1 | grep -c 'BEGIN CERTIFICATE'; sleep .1; done

In ~30% of cases, the chain has 3 certificates, instead of 2:

 0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.actions.githubusercontent.com
 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
 2 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
   i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root

fingerprints:
15:E2:91:08:71:81:11:E5:9B:3D:AD:31:95:46:47:E3:C3:44:A2:31
1C:58:A3:A8:51:8E:87:59:BF:07:5B:76:B7:50:D4:F2:DF:26:4F:CD
FB:20:FA:8A:6A:93:B3:75:F0:54:81:4F:9E:00:27:3E:A5:1A:61:38

vs

 0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.actions.githubusercontent.com
 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA

fingerprints:
15:E2:91:08:71:81:11:E5:9B:3D:AD:31:95:46:47:E3:C3:44:A2:31
69:38:FD:4D:98:BA:B0:3F:AA:DB:97:B3:43:96:83:1E:37:80:AE:A1

Fortunatelly, this is not a problem for OIDC, since Amazon is getting the same cert chain every time. At least from what I have seen. This problem may be ISP/country specific.

Answer 13 · 2022-01-13T16:37:48.000Z

Is there recommended course of action for mitigation? Spent an hour thinking I broke something until I found this thread.

Answer 14 · 2022-01-13T17:33:48.000Z

There is a viable mitigation here: rotating certificates includes the possibility of changing trust chains in the course of that. What Github could do is publish the trust bundles for their expected CAs, users retrieve that bundle, extract the thumbprints for all valid roots. This is a list, so you can provide multiple fingerprints so that you can tolerate getting different CAs per connection while a rotation is happening, since this is a distributed consistency problem that shouldn't be taken for an atomic operation for clients.

Publishing trust bundles is in fact what Google does for the GCP IdP, but here it seems that the AWS resource definition requires not just the trust vector but enumeration of all the certificates used. Trust vector and name really should be sufficient, but Github could still give users relief by publishing the certs with their chains.

This is preferable to having to look at CT to look for all issuance against the host name, retrieving all valid roots and passing those, as CT is a detective rather than a preventive control against incorrect issuance. If you want to do pinning right, which you generally should for an IdP, you need to verify the root of trust for first use, can use that root to pull the bundle thereafter for updates.

You could also pull published trust bundles then look at what's in CT to determine, but this really points to trust qualification being fairly wonky in the OIDC/OAuth world, which generally expect you either to trust a name via whatever WebPKI universe you accept, or you trust a specific key. AWS wants the latter, the sane middle ground is that you want a name and a specific trust vector, such that you can give trust bundles as opposed to individual HTTPS certificate.

Whatever should be done, there's a bunch of complexity that really should be hidden via an IdP datasource provider.

Answer 15 · 2022-01-13T18:01:30.000Z

@buffyg aws allows upto 5 thumbprint . (just a consideration )

Answer 16 · 2022-01-13T18:31:00.000Z

Larger point:

@buffyg aws allows upto 5 thumbprint . (just a consideration )

But the point is that getting all possible server certificates during a rotation while pinning is not something that a client has ready visibility into because you aren't supposed to need to know that. Such are the mechanics of TLS with WebPKI: you don't look up a key or set of keys for a name, you connect to the thing with that DNS name, you retrieve a cert, validate it against your set of trust anchors. Self-selected published trust vectors so that you pin to issuance path rather than individual certificates, that's the most tractable form because it follows the design of TLS+WebPKI, and it still needs publication by a side channel because that's not what JOSE/JWKS provides for. Github as things stand just does plain WebPKI, as best as I can see.

All possible server certificates, that's still potentially tractable by adding a side channel for publishing all certificates that are meant to be in use, but it's not at all how things were designed to work in the first place, is treating WebPKI like self-signed keys which therefore all need to be published, hence the fragility in handling rotations.

Answer 17 · 2022-01-13T18:44:26.000Z

I'm pretty confident I've accurately described the way CA pinning and related trust bundle publication is supported by the GCP IdP because the details are my best recollection of what I got directly from @agl, but I'll tag him here in case I'm mistaken.

Answer 18 · 2022-01-13T20:58:32.000Z

Duplicate of #355.

Answer 19 · 2022-01-13T21:31:19.000Z

Duplicate of #355.

That's a "Pull Request" and this is an "Issue"

Answer 20 · 2022-01-14T00:24:07.000Z

👋🏾 Thanks all for spotting this and sorry about the issue. We've issued a change log here: https://github.blog/changelog/2022-01-13-github-actions-update-on-oidc-based-deployments-to-aws/.

This is something we'll be looking into to ensure this doesn't happen again.

Answer 21 · 2022-01-14T01:11:21.000Z

👋 To expand on Andy's comment, I'd like to answer a few questions raised.

Does anyone know if this was this replaced due to expiration of the old cert, or a security issue? I'd like to know whether to add the new cert in addition to the old one, or should I replace the old thumbprint.

@danieljamesscott This was a routine rotation due to upcoming expiration. As such, you do not need to remove the previous thumbprint, but may do so as it is no longer in use.

How to get new thumbprint in advance? This will break once every year.

@stevie- This issue was caused by the intermediate CA changing in our certificate's trust chain, which we don't expect to necessarily occur during every SSL rotation/renewal. We're investigating here to determine how to avoid this changing when possible, and will ensure adequate communication in the future if it does need to change and user action is required.

Whatever should be done, there's a bunch of complexity that really should be hidden via an IdP datasource provider.

@buffyg Thanks for your thorough analysis, we'll be looking into options here. The AWS documentation implies there may be some options here to reduce the need for customers to update the thumbprint themselves, although our engineering team will need to investigate if they are feasible.

Again, I'd like to apologize for this issue and reiterate that we are taking steps to ensure this doesn't happen again.

Answer 22 · 2022-01-14T04:43:38.000Z

While it's always annoying to have something like this break deployments, I will say it's quite nice that I was able to:

Go to the repository for the actions
Look at the issues
See a resolution (thanks @suzuki-shunsuke!)
See that the maintainers are aware of the issue and finding ways to prevent it in the future

We've all had to deal with support hell and black boxes when it comes to issues and resolutions before. The community, collaboration, and transparency here is refreshing!

Answer 23 · 2022-01-14T07:26:40.000Z

For those using terraform to manage the OIDC provider in AWS:

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
  client_id_list  = ["sts.amazonaws.com"]
}

I really like that solution, but how secure is it, especially in terms of man-in-the-middle attacks?

Answer 24 · 2022-01-14T09:02:16.000Z

For those using terraform to manage the OIDC provider in AWS:

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
  client_id_list  = ["sts.amazonaws.com"]
}

I really like that solution, but how secure is it, especially in terms of man-in-the-middle attacks?

What are the security implications of having the wrong fingerprint? The Idp queries will still go to token.actions.githubusercontent.com ?

Also out of interest why are we querying https://token.actions.githubusercontent.com/.well-known/openid-configuration rather than just https://token.actions.githubusercontent.com to get the cert information?

It works well for me by the way (using both urls), thanks!

Answer 25 · 2022-01-14T16:30:54.000Z

The AWS documentation implies there may be some options here to reduce the need for customers to update the thumbprint themselves, although our engineering team will need to investigate if they are feasible.

@zarenner I've also talked to AWS about making the underling trust ceremony corporate as opposed to per-customer. They already trust a number of IdPs, suggested they do the same with you, the mechanics can be between your engineering orgs.

I suggested using WebPKI issuance pins, that's what AWS says, which is then garbled by Terraform (looking at the AWS docs, it starts out talking about using CA pins, but the Terraform resource docs instead suggest these should be server thumbprints, not CA, "A list of server certificate thumbprints for the OpenID Connect (OIDC) identity provider's server certificate(s)"). The AWS doc, which gives the procedure for relying on root CAs, is https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html and the Terraform resource doc is https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_openid_connect_provider.

I've since been pointed to a write-up from @sleevi on considerations for OIDC FastFed, suggesting that subsetting WebPKI issuance has viability issues and that Private PKI should be used instead. Also should be pointed out that DigiCert has reduced the lifetime of their ICAs precisely to discourage pinning via ICA: https://knowledge.digicert.com/alerts/DigiCert-ICA-Update.html

Sleevi's notes are here: https://docs.google.com/document/d/e/2PACX-1vQQ-FhNjW0ZhZVTnK1VG_87IBNZKBaJmweYZb1VBRdQCMAWXekqKfxdNl8wL6FWFDJ9pxXxpr66-GZp/pub

Answer 26 · 2022-01-14T19:20:12.000Z

@zarenner mentioned wanting to explore ways to avoid having the issuing CA change. To echo and emphasize @buffyg’s comment, the WebPKI doesn't provide that guarantee, somewhat intentionally.

DigiCert has been explicit about this, and it’s a huge credit to them for implementing organizational changes to help surface these risks during routine situations (such as rotations), where folks are aware and can respond to consequences, because the alternative is these sorts of things get surfaced during non-routine situations, like responses to CA compromise or distrust.

A big part of this philosophy was seeing how long it took organizations to successfully transition off certificates from DigiNotar, WoSign/StartCom, and Symantec, three CAs that demonstrated significant security control failures. For an unfortunately large number of users, they had trouble rotating certificates precisely because they had end users who had pinned.

Beyond such cases, though, the WebPKI ecosystem has intentionally been moving to shorter (effective) lifetimes to root certificates. When you use a 10-year old root, for example, you also inherit 10 years of any mistakes it has made potentially causing harm to your organization. The old Verizon PKI demonstrated this best: Verizon did not even know everyone they had issued sub-CAs to, any of which could have maliciously issued certificates that anyone who trusted the Verizon roots would accept. DigiCert, when they acquired Verizon, spent an incredible amount of time investigating and revoking those certificates, because that’s a huge risk.

As one of the authors of RFC 7469, and as one who helped drive removing support from Chrome, I’ve definitely been trying to help educate folks about the dangers of pinning. Since we’re engineers here, the way I would describe Certificate pinning is a bit like reaching into an opaque C struct: you’re reaching into implementation details that aren’t ABI or API guaranteed.

That is, if you had a foo.h that had

typedef struct FOO_internal FOO;

then it would be inherently unsafe in your libraries bar.c to do

void baz(FOO* foo) {
  *((int*)((char*)(foo) + 12)) = 42;
}

because you’re reaching in to the internal implementation details and “violating” the API contract.

Pinning to the WebPKI is basically that: reaching into implementation details of the trust graph that, by design, aren’t guaranteed to be stable. If you need a stable API, then private PKI let’s you define those contracts as you need.

Hopefully this doesn’t come off as a lecture 😅 It’s just very much “here be dragons”, and any pinning to the Web PKI is going to (eventually) explode, because just like software itself needs to get regularly updated to respond to security risks, so does the WebPKI.

Note: while Google does publish a root bundle at https://pki.goog, that’s not really for encouraging pinning, and you’ll note it’s a broad selection of CAs. However, even with that broad selection, there have still been operational issues where Google ends up getting a certificate outside of that set, and interoperability issues can come up. So publishing a set of trust anchors isn’t much of a solution: it’s primarily there to ensure at least a minimum set or “everyone Google may decide to do business with” is included.

Answer 27 · 2022-01-16T20:36:36.000Z

Small adjustment to the terraform snippet provided earlier... This will use all the certificates instead of only the 0-index...

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  thumbprint_list = data.tls_certificate.github.certificates[*].sha1_fingerprint
  client_id_list  = ["sts.amazonaws.com"]
}

Answer 28 · 2022-01-16T21:14:47.000Z

Small adjustment to the terraform snippet provided earlier... This will use all the certificates instead of only the 0-index...

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  thumbprint_list = data.tls_certificate.github.certificates[*].sha1_fingerprint
  client_id_list  = ["sts.amazonaws.com"]
}

I'm not sure this is advisable, I might be wrong but I think that, at best, it's useless if the target only checks the end certificate of the provider, at worst, it makes the target trust any certificates signed by the same authority GitHub is using.

Answer 29 · 2022-01-16T22:15:16.000Z

I'm not sure this is advisable, I might be wrong but I think that, at best, it's useless if the target only checks the end certificate of the provider, at worst, it makes the target trust any certificates signed by the same authority GitHub is using.

Oh that may be. I'm not sure. I suppose I saw multiple fingerprints and figured that must be a mechanism to support rotation, such that both current and <next> would work for some period of time.

Answer 30 · 2022-01-16T22:32:01.000Z

Here's the output of the data source... I'm certainly not an expert on TLS or OIDC. Happy to defer to others that are more knowledgeable.

    "certificates" = tolist([
      {
        "is_ca" = true
        "issuer" = "CN=DigiCert Global Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US"
        "not_after" = "2030-09-23T23:59:59Z"
        "not_before" = "2020-09-24T00:00:00Z"
        "public_key_algorithm" = "RSA"
        "serial_number" = "13567650854749339296468135199911180260"
        "sha1_fingerprint" = "6938fd4d98bab03faadb97b34396831e3780aea1"
        "signature_algorithm" = "SHA256-RSA"
        "subject" = "CN=DigiCert TLS RSA SHA256 2020 CA1,O=DigiCert Inc,C=US"
        "version" = 3
      },
      {
        "is_ca" = false
        "issuer" = "CN=DigiCert TLS RSA SHA256 2020 CA1,O=DigiCert Inc,C=US"
        "not_after" = "2023-01-11T23:59:59Z"
        "not_before" = "2022-01-11T00:00:00Z"
        "public_key_algorithm" = "RSA"
        "serial_number" = "11052166568250529137901871408801069995"
        "sha1_fingerprint" = "15e29108718111e59b3dad31954647e3c344a231"
        "signature_algorithm" = "SHA256-RSA"
        "subject" = "CN=*.actions.githubusercontent.com,O=GitHub\\, Inc.,L=San Francisco,ST=California,C=US"
        "version" = 3
      },
    ])

Answer 31 · 2022-01-17T11:12:10.000Z

See also https://stackoverflow.com/questions/69247498/how-can-i-calculate-the-thumbprint-of-an-openid-connect-server/69247499#69247499 as to how to calculate the thumbprint

Answer 32 · 2022-07-13T19:22:25.000Z

If you're looking for a one-liner to find the latest thumbprint, here's one I came up with based on the IAM docs:

openssl s_client \
        -servername token.actions.githubusercontent.com \
        -showcerts \
        -connect token.actions.githubusercontent.com:443 2>/dev/null < /dev/null | \
    openssl x509 -fingerprint -noout | \
    grep Fingerprint | \
    tr -d ':' | \
    tr '[:upper:]' '[:lower:]' | \
    cut -f2 -d=

Answer 33 · 2022-08-25T12:31:44.000Z

Did anything change yesterday? I have been having issues deploying, and I tried above, which did give me a different thumbprint: 15e29108718111e59b3dad31954647e3c344a231

Error I'm getting in AWS:
Resource handler returned message: "OpenIDConnect Provider not found for arn arn:aws:iam:::oidc-provider/token.actions.githubusercontent.com (Service: Iam, Status Code: 404, Request ID: d48f1da1-2f86-472e-86c9-1207c1323d6a, Extended Request ID: null)" (RequestToken: 6a0eba58-6fda-99ba-f879-b1351d933b40, HandlerErrorCode: NotFound)

Answer 34 · 2022-08-25T13:09:41.000Z

Looks like the new fingerprint worked after I quoted it in "yaml".... Something annoying did happen yesterday and today. Todays error was much better and same thing that happened months ago. Yesterdays was confusing and wasted lots of time.

Bad error: No OpenIDConnect provider found in your account for https://token.actions.githubusercontent.com/
Good error: OpenIDConnect provider’s HTTPS certificate doesn’t match configured thumbprint

Answer 35 · 2022-10-05T22:06:18.000Z

Thank you everyone for the thorough discussion here, this issue has died down a lot since it initially happened, so I don't think we need to keep it open anymore. Hopefully this situation won't happen again 🙂

Answer 36 · 2022-10-05T22:06:41.000Z

⚠️Comment Visibility Warning⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Answer 37 · 2023-06-24T06:43:08.000Z

Error: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

I am using thumbprint : 6938fd4d98bab03faadb97b34396831e3780aea1

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::XXXXXXXXXXX:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": [
                "sts:AssumeRoleWithWebIdentity"
            ],
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:sub": "repo:XXXXX/*"
                },
                "ForAllValues:StringEquals": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
                    "token.actions.githubusercontent.com:iss": "https://token.actions.githubusercontent.com"
                }
            }
        }
    ]
}

type: OpenID Connect
provider: https://token.actions.githubusercontent.com
thumbprint : 6938fd4d98bab03faadb97b34396831e3780aea1
Audience: sts.amazonaws.com

Answer 38 · 2023-06-27T00:19:38.000Z

We are also seeing the same issue as @kamranNBS

Answer 39 · 2023-06-27T00:24:24.000Z

Seems like GH updated their SSL certificate. You should go to IAM identity providers, find github connect and add new fingerprint (f879abce0008e4eb126e0097e46620f5aaae26ad):

PS: Huge thanks @micahhausler for the one-liner

Answer 40 · 2023-06-27T00:24:59.000Z

Hi, I am experiencing the same error as @kamranNBS

OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

Updating to aws-actions/configure-aws-credentials@v2 did not help

Answer 41 · 2023-06-27T00:27:29.000Z

Confirming @ixti's solution worked 😄

Answer 42 · 2023-06-27T00:28:19.000Z

Thanks, @ixti - this fixed it 🥳

Out of interest, where did you find the new fingerprint?

EDIT: saw his comment thanking @micahhausler using this

Answer 43 · 2023-06-27T00:38:07.000Z

did the order of the certs change? the fingerprints of the top/bottom certs are as follows

% openssl x509 -in top.crt -fingerprint -sha1 -noout
SHA1 Fingerprint=F8:79:AB:CE:00:08:E4:EB:12:6E:00:97:E4:66:20:F5:AA:AE:26:AD

% openssl x509 -in bottom.crt -fingerprint -sha1 -noout
SHA1 Fingerprint=69:38:FD:4D:98:BA:B0:3F:AA:DB:97:B3:43:96:83:1E:37:80:AE:A1

I think the actual certificates are the same.

Answer 44 · 2023-06-27T01:44:41.000Z

did the order of the certs change? the fingerprints of the top/bottom certs are as follows

Dunno, but that's the same what happened a year ago:

Answer 45 · 2023-06-27T02:14:43.000Z

Glad to see I am not the only one who suddenly got this error 😅 Is there not some way for us to automate the thumbprint? Kinda hectic that all your workflows just fall over and you have to go manually change the thumbprint 🤔

Answer 46 · 2023-06-27T04:18:23.000Z

Its giving me this now Not authorized to perform sts:AssumeRoleWithWebIdentity

however my trust relation looks likes,

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::XXXXXXXXXXX:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": [
                "sts:AssumeRoleWithWebIdentity"
            ],
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:sub": "repo:XXXXX/*"
                },
                "ForAllValues:StringEquals": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
                    "token.actions.githubusercontent.com:iss": "https://token.actions.githubusercontent.com"
                }
            }
        }
    ]
}

Answer 47 · 2023-06-27T08:35:00.000Z

something strange is happening just now... run from 1hr ago (thumb generated via terraform data)

        id              = "arn:aws:iam::***:oidc-provider/token.actions.githubusercontent.com"
        tags            = {}
      ~ thumbprint_list = [
          - "6938fd4d98bab03faadb97b34396831e3780aea1",
          + "1c58a3a8518e8759bf075b76b750d4f2df264fcd",
        ]

Answer 48 · 2023-06-27T13:27:26.000Z

I am using the AWS guide to verify the thumbprint and when I request the certificates from GitHub with:

openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443

I get two certificates, the first certificate is always f879abce0008e4eb126e0097e46620f5aaae26ad but the second one randomly changes between 6938fd4d98bab03faadb97b34396831e3780aea1 and 1c58a3a8518e8759bf075b76b750d4f2df264fcd.

Answer 49 · 2023-06-27T16:28:17.000Z

I get two certificates, the first certificate is always f879abce0008e4eb126e0097e46620f5aaae26ad but the second one randomly changes between 6938fd4d98bab03faadb97b34396831e3780aea1 and 1c58a3a8518e8759bf075b76b750d4f2df264fcd.

The second is the DigiCert CA, one of them ends in 2030, the other is in 2031. This is the one with 1c58a3a8518e8759bf075b76b750d4f2df264fcd fingerprint:

 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 14 00:00:00 2021 GMT; NotAfter: Apr 13 23:59:59 2031 GMT

This is the one with 6938fd4d98bab03faadb97b34396831e3780aea1 fingerprint:

 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep 24 00:00:00 2020 GMT; NotAfter: Sep 23 23:59:59 2030 GMT

Answer 50 · 2023-06-27T16:30:20.000Z

Either way I believe the correct fingerprint to be used is: f879abce0008e4eb126e0097e46620f5aaae26ad

Answer 51 · 2023-06-27T20:29:45.000Z

Do we need to change the thumbprint on existing deployments?

Answer 52 · 2023-06-27T20:31:44.000Z

Last week (we started noticing this last week) we had to resort to the following in Terraform but even that does not work. We currently have 3 thumbprints

6938fd4d98bab03faadb97b34396831e3780aea1
f879abce0008e4eb126e0097e46620f5aaae26ad
1c58a3a8518e8759bf075b76b750d4f2df264fcd

and none of them are working. This appears to be transient because it was really bad a week ago, went away for a while and is back today

data "tls_certificate" "github_actions_oidc_endpoint" {
  url = "https://token.actions.githubusercontent.com"
}

locals {
  # GitHub's root cert: DigiCert TLS RSA SHA256 2020 CA1
  root_cert_thumbprint = "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  # Whenever GitHub updates its cert list, we'll have to re-apply the TF module
  thumbprints = concat(data.tls_certificate.github_actions_oidc_endpoint.certificates[*].sha1_fingerprint, [local.root_cert_thumbprint])
}

resource "aws_iam_openid_connect_provider" "github_oidc" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = local.thumbprints
}

UPDATE: Seems like we had not applied this change to all the places we needed to. I think there are a bunch of other similar solutions in this thread that may yield better results. Honestly skeptical of the value of even codifying dynamically in IaC given it'll still be broken until you re-run it, at that point maybe just better to hardcode them 🤷🏼

Answer 53 · 2023-06-27T20:34:02.000Z

Reopening this because of reported issues. We publish some documentation on how to verify OIDC endpoint certificates so let's follow that.

The jwks_uri that GitHub gives us is https://token.actions.githubusercontent.com/.well-known/jwks, taken from https://token.actions.githubusercontent.com/.well-known/openid-configuration
Let's look at what GitHub is giving us here (thanks to https://raymii.org/s/articles/OpenSSL_-_Get_all_certificates_from_a_website_in_plain_text.html for an easy one liner):

OLDIFS=$IFS; IFS=':'
certificates=$(openssl s_client -connect token.actions.githubusercontent.com:443 \
             -showcerts -tlsextdebug 2>&1 </dev/null | \
             sed -n '/-----BEGIN/,/-----END/ {/-----BEGIN/ s/^/:/; p}'); 
for certificate in ${certificates#:}; do
  echo $certificate | openssl x509 -noout -subject -dates;
  echo $certificate | openssl x509 -noout -fingerprint;
done;
IFS=$OLDIFS

On my local machine, I get

subject=C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.actions.githubusercontent.com
notBefore=Nov  4 00:00:00 2022 GMT
notAfter=Nov  7 23:59:59 2023 GMT
SHA1 Fingerprint=F8:79:AB:CE:00:08:E4:EB:12:6E:00:97:E4:66:20:F5:AA:AE:26:AD
subject=C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
notBefore=Sep 24 00:00:00 2020 GMT
notAfter=Sep 23 23:59:59 2030 GMT
SHA1 Fingerprint=69:38:FD:4D:98:BA:B0:3F:AA:DB:97:B3:43:96:83:1E:37:80:AE:A1

IAM requires that the thumbprint for the top intermediate CA that signed the certificate is the one configured. That would be the certificate signed by DigiCert, with a fingerprint of 6938FD4D89BAB03FAADB97B34396831E3780AEA1. This is the same thumbprint that GitHub says to use for AWS OIDC. I think GitHub doesn't intend to change this considering the churn that happened last time.'

I'm not sure what's going on, but the reports by @dabbeg and @ixti seem to indicate that GitHub is giving out different intermediates, probably depending on what global endpoint you're hitting. We'll reach out to some folks on the GitHub side to see if we can find if anything is going on.

In the meantime, our recommendation is to always manually verify the intermediate thumbprint you're getting from the OIDC endpoint. You can do that with the one/some liners I've used here, or the IAM documentation I've linked above. Once you've verified your endpoint thumbprints, you should be safe to add them to your IdP configuration.

Answer 54 · 2023-06-27T20:36:28.000Z

@kellertk 100% but I seem to have all the possible fingerprints set and still getting issues, unsure about others

Answer 55 · 2023-06-27T20:38:26.000Z

I think this was just posted: https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/, looks like there are multiple thumbprints now, as we suspected. I will update our documentation.

Answer 56 · 2023-06-27T20:45:27.000Z

Nice of them to give us plenty of warning on this.

Answer 57 · 2023-06-27T20:54:05.000Z

I had hardcoded the value before in my TF - this is what I just wrote to make it more dynamic.

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github-actions" {
  url = "https://token.actions.githubusercontent.com"
  client_id_list = [
    "sts.amazonaws.com"
  ]

  thumbprint_list = [for certificate in data.tls_certificate.github.certificates : certificate.sha1_fingerprint]
}

What is interesting is that it totally removed the one from Jan 2022:

  ~ resource "aws_iam_openid_connect_provider" "github-actions" {
        id              = "arn:aws:iam::NNNNNNNN:oidc-provider/token.actions.githubusercontent.com"
        tags            = {}
      ~ thumbprint_list = [
          - "6938fd4d98bab03faadb97b34396831e3780aea1",
            "1c58a3a8518e8759bf075b76b750d4f2df264fcd",
          + "f879abce0008e4eb126e0097e46620f5aaae26ad",
        ]
        # (4 unchanged attributes hidden)
    }

Which seems to jive with what people were seeing above.

Answer 58 · 2023-06-27T21:01:40.000Z

The implication of GitHub's blog post is that you cannot use data.tls_certificate because they may serve you 1 of those 2 different CA certificates at random. A simple test configuration to prove that:

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

output "certificates" {
  value = [
    for cert in data.tls_certificate.github.certificates : {
      is_ca = cert.is_ca
      sha1_fingerprint = cert.sha1_fingerprint
    }
  ]
}

terraform {
  required_providers {
    tls = {
      source = "hashicorp/tls"
    }
  }
}

If you run this in a loop, sometimes you get this:

certificates = [
  {
    "is_ca" = true
    "sha1_fingerprint" = "6938fd4d98bab03faadb97b34396831e3780aea1"
  },
  {
    "is_ca" = false
    "sha1_fingerprint" = "f879abce0008e4eb126e0097e46620f5aaae26ad"
  },
]

And other times:

certificates = [
  {
    "is_ca" = true
    "sha1_fingerprint" = "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  },
  {
    "is_ca" = false
    "sha1_fingerprint" = "f879abce0008e4eb126e0097e46620f5aaae26ad"
  },
]

Answer 59 · 2023-06-27T21:04:11.000Z

Is there any way to fully automate this?

Answer 60 · 2023-06-27T21:21:41.000Z

ok - I have a reasonable enough solution that I am going to go with...

This code is subideal in that there are hard coded values in there but given the fact that there is no way to be handed a list of all known thumbprints with a single call this is the best I can come up with until someone makes a module (hint hint AWS). If you run this it will set the world to a known state and likely never change again until you are unlucky enough to get a new thumbprint - even this code will not succeed 100% of the time getting a complete list but you are much better off running this than not running it as it has a non-zero chance of showing the change over enough runs.

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github-actions" {
  url = "https://token.actions.githubusercontent.com"
  client_id_list = [
    "sts.amazonaws.com"
  ]

  # Add any _known_ thumbprints here. If they do change you will need to update.
  # At least this will capture any new ones that might be added.
  thumbprint_list = distinct(
    concat(
      [
        "6938fd4d98bab03faadb97b34396831e3780aea1",
        "1c58a3a8518e8759bf075b76b750d4f2df264fcd",
      ],
      [for certificate in data.tls_certificate.github.certificates : certificate.sha1_fingerprint if certificate.is_ca]
    )
  )
}

In my perfect world this list would be curated by those that know - Either AWS or GH in the form of a TF module or at the very least an HTTPS endpoint with the data in a nice format.

Answer 61 · 2023-06-27T22:54:01.000Z

In the ideal world, I like to see GitHub as identity provider that built-in into AWS IAM, similar to Facebook / Google / Cognito:

"Principal":{"Federated":"cognito-identity.amazonaws.com"}
"Principal":{"Federated":"www.amazon.com"}
"Principal":{"Federated":"graph.facebook.com"}
"Principal":{"Federated":"accounts.google.com"}

This will eliminate the need to configure separate IdP in IAM.

Answer 62 · 2023-06-27T23:36:40.000Z

The JWKS endpoint is discoverable/well-known, and that returns the signing certificates, which OIDC providers can used to verify material those certificates signed, such as GitHub's ID tokens. But the certificates contained in there have a subject name of vsts-vstsghrt-gh-vso-oauth.visualstudio.com and a shorter expiration. They are not what we're after here.

The certificates in question here are the CA certificates that GitHub uses to secure HTTPS traffic to https://token.actions.githubusercontent.com. Providing thumbprints ensures that even if someone managed to hijack the githubusercontent.com domain, AWS will only transmit token material for identification if it can verify that the server is actually the one it expects, via the thumbprint.

The natural place for this information would seem to be the meta API:

https://docs.github.com/en/rest/meta/meta

You can use this API today to pre-establish SSH trust with GitHub by adding the public keys returned there:

https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints

If you could discover the Actions OIDC server's CA certificate fingerprints from the meta API, you could do the same here. Right now, it is impossible to automate this reliably as there is no machine-readable way to get this information deterministically.

Answer 63 · 2023-06-28T07:43:16.000Z

Github promised last year they wouldn't break this again https://github.blog/changelog/2022-01-13-github-actions-update-on-oidc-based-deployments-to-aws/

This seems like a regression.

Answer 64 · 2023-06-28T08:06:34.000Z

When checking the certificate chain for token.actions.githubusercontent.com, I'm still seeing randomization in the intermediate certificates being served, which @dabbeg and @ixti reported earlier.

I cross referenced the differing expiry dates via DigiCert, and it seems that the the DigiCert Trusted G4 TLS RSA SHA384 2020 CA1 (thumbprint 9f90f8c8664b09959c2909ac9a78aea8e26d88e8) should also be included, although GitHub didn't reference it in their latest blog post.

Answer 65 · 2023-06-28T08:18:38.000Z

Note that technically AWS only pins the jwks_uri and the jwks_uri could be a different domain than the openid-configuration file. Not that I think the domain of jwks_uri will change. But better be safe than sorry. If you want to be pedantically correct you should do this:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

locals {
  issuer     = "token.actions.githubusercontent.com"
  issuer_url = "https://${local.issuer}"
}

data "http" "github_openid_configuration" {
  request_headers = { Accept = "application/json" }
  url             = "${local.issuer_url}/.well-known/openid-configuration"
}

locals {
  # The jwks_uri is not guaranteed to be the same domain as the issuer_url
  # according to OIDC and AWS pins to the jwks_uri
  # https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
  jwks_uri = jsondecode(data.http.github_openid_configuration.body).jwks_uri
}

data "tls_certificate" "github" {
  url = local.jwks_uri
}

resource "aws_iam_openid_connect_provider" "github" {
  url            = local.issuer_url
  client_id_list = ["sts.amazonaws.com"]
  thumbprint_list = distinct([
    # Per https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/
    "6938fd4d98bab03faadb97b34396831e3780aea1",
    "1c58a3a8518e8759bf075b76b750d4f2df264fcd",
    data.tls_certificate.github.certificates[0].sha1_fingerprint,
  ])
}

Answer 66 · 2023-06-28T15:01:20.000Z

There is little GitHub can do to avoid this breaking again and again, this is AWS fault: #669

GitHub, and any OIDC provider, should be free to deploy any TLS certificate chains they wanted without notifying their users.

Is there any way to fully automate this?

@jensenbox push AWS to solve that "Internal P82323250" issue. No automation should be needed when a CA trust chain is in place.

jwks_uri could be a different domain than the openid-configuration

Also @arianvp is right, all this terraform automations only work accidentally. But jwks_uri's domain doesn't necessarily match openid-configuration's domain.

Answer 67 · 2023-06-28T15:16:52.000Z

Its giving me this now Not authorized to perform sts:AssumeRoleWithWebIdentity

@kamranNBS were you able to fix this? I am getting the same

Answer 68 · 2023-06-28T15:45:40.000Z

There is little GitHub can do to avoid this breaking again and again, this is AWS fault: #669

GitHub, and any OIDC provider, should be free to deploy any TLS certificate chains they wanted without notifying their users.

Is there any way to fully automate this?

@jensenbox push AWS to solve that "Internal P82323250" issue. No automation should be needed when a CA trust chain is in place.

@kellertk Apologies if you're not the right person to ask but do you have any insight on the status of this internal ticket?

Answer 69 · 2023-06-28T16:45:25.000Z

The developers of this action won't be able to provide any insights or statuses for that ticket, but our stance is that we hope a permanent solution will get done before the next time this happens

Answer 70 · 2023-06-28T19:03:25.000Z

For those using terraform to manage the OIDC provider in AWS:

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
  client_id_list  = ["sts.amazonaws.com"]
}

Thanks for this suggestion, but this only grabs the first returned Well Known intermediary. Per this GitHub blog post, you must trust them all (currently 2), or your connection could be unstable.

Suggestion:

locals {
  github_cert_thumbprints = [for cert in data.tls_certificate.github.certificates : cert.sha1_fingerprint]
}

data "tls_certificate" "github" {...}

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  thumbprint_list = local.github_cert_thumbprints
  client_id_list  = ["sts.amazonaws.com"]
}

Answer 71 · 2023-06-28T19:44:06.000Z

Reiterating #357 (comment), it's not a question of certificates[0].sha1_fingerprint or certificates[*].sha1_fingerprint or any variation on that theme. The server presents different CA certificates, at random, presumably depending on which backend you reach.

If you terraform apply the above configuration in a loop, it will flap depending on which intermediate GitHub's server presents when Terraform opens the TLS connection.

A search for the new fingerprint turns up a result in the Azure CA index so presumably this behavior is inherited from some Azure infrastructure in use behind the scenes.

The example above that uses the http provider to obtain the well-known configuration and then makes a TLS connection to the jwks_uri in the response is the most correct way to implement AWS's instructions for fingerprinting an OIDC provider in Terraform. In theory you can filter the certificates on is_ca but in practice GitHub is only returning one at a time and CA certificates should come first so there's no actual difference today.

The issue here is that there are 2 different intermediate certificates in use, depending on the request, both issued from the same root. And because AWS does not trust the root and requires users to explicitly trust the intermediates directly presented by the server, you end up with tight coupling and these breakage incidents.

Answer 72 · 2023-06-28T22:04:50.000Z

This topic is a big blame on GitHubs architects side as they should be aware of how their choice of infrastructure will impact a critical part of the authentication mechanism implemented by ANY OIDC CAPABLE system. I'd expect at least GitHub to know what a contract is and what to offer when you reach "well known" endpoint...

Answer 73 · 2023-06-29T05:17:12.000Z

After adding the new thumbprint, I now have the following error: Error: Not authorized to perform sts:AssumeRoleWithWebIdentity

It was all working fine before the thubprint issue and the same setup works on another aws account after the thumbprint update. Here's the trust policy used:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:myOrg/myRepo:*"
                },
                "StringEquals": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Any idea?

Answer 74 · 2023-06-29T05:20:23.000Z

@cpp-cat You'll have to add multiple thumbprints. I am using terraform to setup the Idp on AWS and by adding this it worked for me:

  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1",
    "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  ]

It looks like the OIDC mechanism expects a different thumbprint depending on the AWS account ID or region. Not sure tho!

Answer 75 · 2023-06-29T05:25:22.000Z

@ahasna Thanks for the reply. I have 3 thumbprints:

The fact that I now have a different error, does it mean I got pass the thumbprint validation and now it's trying the assume the role and it fails when applying the conditions?

Answer 76 · 2023-06-29T06:50:20.000Z

@cpp-cat Are you setting the token permissions correctly for your Job?

    permissions:
      contents: read
      id-token: read

Answer 77 · 2023-06-29T08:06:09.000Z

Github Official thumbrpint announcement https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/

Answer 78 · 2023-06-29T08:21:50.000Z

@adudek blame is on AWS, not GitHub: #357 (comment)

Not trusting TLS certificate chains issued by CAs and requiring users to hardcode fingerprints of certificates they don't operate is poor design by AWS.

Answer 79 · 2023-06-29T08:51:25.000Z

@n1ngu by this extension, blame all OIDC implementers that are not willing to support github out of the box, just because github doesn't understand contracts nor it provides means to update those automatically. Do I understand correctly, that we expect every vendor to read github announcements and copy-paste its configuration to their solution, also bending their release cycles to satisfy githubs lack of insight?

Answer 80 · 2023-06-29T09:21:28.000Z

Technically GitHub did nothing wrong here. Nowhere does OIDC spec mandate pinning to intermediate TLS certificates.
But it's annoying that GitHub was aware of the problem and broke it anyway without heads up.

blame all OIDC implementers that are not willing to support github out of the box,

We should. because if they're doing pinning they're not following OIDC by the letter. OIDC actually says jwks_uri can change at any time. Pinning the intermediate cert of jwks_uri is really bad and not compliant with the spec at all

Answer 81 · 2023-06-29T09:56:42.000Z

blame all OIDC implementers that are not willing to support github out of the box,

We should. because if they're doing pinning they're not following OIDC by the letter. OIDC actually says jwks_uri can change at any time. Pinning the intermediate cert of jwks_uri is really bad and not compliant with the spec at all

I'm willing to accredit blame to AWS if GitHub will provide means to automatically supply all required OIDC information (yes, all certificates in use) and if it will not rely on reading their announcements. In the current situation even if AWS wanted - it could not implement it properly, because someone needs to be informed that the change was made. In this scenario, all blame would be put on AWS. Is it 90's again or what?

Answer 82 · 2023-06-29T10:07:36.000Z

I'm willing to accredit blame to AWS if GitHub will provide means to automatically supply all required OIDC information (yes, all certificates in use) and if it will not rely on reading their announcements.

That would be necessary if GitHub was deploying self-signed certificates. The whole point of using CA certificates is not having to announce certificate rotations. If AWS trusted CA certificates none of this would happen. And this is unrelated to OIDC specs.

Answer 83 · 2023-06-29T10:09:28.000Z

blame all OIDC implementers that are not willing to support github out of the box,

We should. because if they're doing pinning they're not following OIDC by the letter. OIDC actually says jwks_uri can change at any time. Pinning the intermediate cert of jwks_uri is really bad and not compliant with the spec at all

I'm willing to accredit blame to AWS if GitHub will provide means to automatically supply all required OIDC information (yes, all certificates in use) and if it will not rely on reading their announcements. In the current situation even if AWS wanted - it could not implement it properly, because someone needs to be informed that the change was made. In this scenario, all blame would be put on AWS. Is it 90's again or what?

GitHub already publishes all the information needed programatically and in compliance with OIDC. Namely PKI. their jwks_uri is signed by a globally trusted root CA. AWS should perform TLS verification like everyone else in the world and not come up with alternative schemes like pinning intermediate certificates.

Only thing AWS needs to do is verify the certificate chain as per the TLS spec. That they don't do this is extremely weird.

Answer 84 · 2023-06-29T11:25:32.000Z

Only thing AWS needs to do is verify the certificate chain as per the TLS spec. That they don't do this is extremely weird.

I can only assume it is because of services like EKS that use self-signed certificates for their control plane and OIDC endpoints.

Answer 85 · 2023-06-29T11:43:38.000Z

GitHub already publishes all the information needed programatically and in compliance with OIDC. Namely PKI. their jwks_uri is signed by a globally trusted root CA. AWS should perform TLS verification like everyone else in the world and not come up with alternative schemes like pinning intermediate certificates.

What if PKI Consortium decides to host (part of) its services with AWS? Would You trust this solution? I'd trust only pinned information that was vetted.

Answer 86 · 2023-06-29T12:01:37.000Z

How are you suggesting to retrieving that pin from GitHub? Through TLS. And we're back to square one. WebPKI is safe. Certificate Transparency log makes abuse obvious. Pinning is unnecessary complexity.

Answer 87 · 2023-06-29T15:37:30.000Z

Hi everyone, let's try and keep the discussion here related to this action only. The way thing are today:

If you want to use OIDC to authenticate to GitHub, you will see one of multiple intermediate certificates.
Instead of verifying the certificate chain by trusting a root anchor, AWS IAM instead verifies the federated identity provider by statically configuring the intermediate's thumbprint. As far as I am aware this requirement is unique to AWS.
IAM will also attempt to detect the thumbprint during set up, but this is incompatible with the concept of multiple correct thumbprints. You therefore need to do additional configuration in order to trust all of them, which as of my writing is these two: 6938fd4d98bab03faadb97b34396831e3780aea1, 1c58a3a8518e8759bf075b76b750d4f2df264fcd.

The path forward to making this a better experience involves changing how IAM works, or how GitHub is managing their certificates, or both. I don't have any information to share today other than the IAM team is aware of this sharp edge and we want to make it better in the future. If I have anything to announce there, I'll share that as soon as possible.

We've updated the readme and the example CloudFormation template for this action's documentation. I'd also note that if you're using the CDK OpenIdConnectProvider construct, you will need to manually configure thumbprints as well. If anyone has any suggestions on how to make our documentation and setup process clearer, we'd be happy to take that feedback.

Answer 88 · 2023-06-29T20:33:26.000Z

I had updated our OIDC provider with the new fingerprints but still ran into the same issue.

Don't forget to update not only the OIDC provider in the account where the runners are hosted, but also in other accounts from which roles are being assumed.

Answer 89 · 2023-06-30T14:52:14.000Z

1c58a3a8518e8759bf075b76b750d4f2df264fcd

Thanks! Works for me

Answer 90 · 2023-07-01T16:08:15.000Z

1c58a3a8518e8759bf075b76b750d4f2df264fcd

Confirmed this also worked for me, I have both 6938fd4d98bab03faadb97b34396831e3780aea1 and 1c58a3a8518e8759bf075b76b750d4f2df264fcd as thumbprints now

Answer 91 · 2023-07-06T09:12:47.000Z

I am using a GitHub self-hosted runner in an AWS AutoScaling Group (ASG). The IAM Role that is used has both thumbprints.

6938fd4d98bab03faadb97b34396831e3780aea1
1c58a3a8518e8759bf075b76b750d4f2df264fcd

But I was still receiving the error below.

OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

I had to recreate the EC2 Instance by scaling down and up the ASG to get the error to go away. Has anyone else experienced this issue with self-hosted runners?

Answer 92 · 2023-07-06T14:15:35.000Z

@hxrsmurf I saw the exact same error persist for a few jobs on the GitHub hosted runners even though I had the full history or thumbprints configured for OIDC:

a031c46782e6e6c662c2c87c76da9aa62ccabd8e # Original GitHub cert
6938fd4d98bab03faadb97b34396831e3780aea1 # As of 13/01/2022
f879abce0008e4eb126e0097e46620f5aaae26ad # As of 01/07/2023
1c58a3a8518e8759bf075b76b750d4f2df264fcd # As of 01/07/2023

See:

Note that it was only a few jobs out of many and some of the jobs were in workflows for which there were other successful jobs.

The errors have only stopped in the last couple of days (not exactly sure when as I left it for a while). Strangely all the errors were from jobs targeting us-east-1 though it was only a few jobs so a small sample size (and there were other successful jobs for us-east-1).

All I can say is all my jobs on GitHub hosted runners are now working but there were peristent errors for a few days despite having all thumbprints configured.

Answer 93 · 2023-07-07T23:27:18.000Z

Hello! @wellsiau-aws beat me to announcing this slightly, but I have some changes to share: the IAM team have released an update for thumbprint configuration for GitHub (and only GitHub at this time) that will permanently solve this issue!

As of July 6, 2023, it is no longer necessary to specify a specific thumbprint in your IAM Identity Provider configuration. AWS will secure communication with tokens.actions.githubusercontent.com using our library of trusted CAs rather than using a certificate thumbprint. You should see a message in your AWS account if you look at the configured identity provider confirming that this is the case.

The API still appears to require that some thumbprint is configured, since that will require other API and UI changes, but your thumbprint will be completely ignored and OIDC will still work. The message in the IAM console will persist to remind you of this. I successfully tested this out in my own AWS account by setting my configured thumbprint to afafafafafafafafafafafafafafafafafafafaf and was able to get authenticated.

Note: this change will not affect GHES customers, since they have a different hostname and issuer from their IdP.

You should not need to make any changes to your existing configuration with this change. Please open up a separate issue in this repo if you're continuing to have problems with the OIDC configuration, or reach out to AWS Support.

Answer 94 · 2023-07-07T23:27:42.000Z

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Answer 95 · 2023-07-08T17:43:31.000Z

It would be lovely to get screenshots of the before and after in the console. Just so we know what to expect.

This sounds like an amazing improvement.

How does this affect the terraform side of the world?

Answer 96 · 2023-07-08T19:07:43.000Z

@jensenbox here's the after. I removed one of the fingerprints, and overwrote the other with the thumbprint you see. Still works!