Error in GeneralNames while generating a certificate with AuthorityKeyIdentifier

Question

Error in GeneralNames while generating a certificate with AuthorityKeyIdentifier

ricardo-reis-1970 opened this issue 6 months ago · 9 comments

ricardo-reis-1970 commented 6 months ago

Environment

I'm working on:

Ubuntu 24.04
Python 3.12.3
asn1crypto 1.5.1

Issue

I'm using asn1crypto to decode certificates. Moving forward, I'll be using it to also encode certificates. In order to test the encoding sanity, I'm trying to encode existing certificates from my system, using the following cycle:

decode / load a certificate from file;
get the data out of asn1crypto into my own data structures / class hierarchy;
3. rebuild the certificate back from data;
dump the certificate to file;
compare files;
open champagne.

I'm currently stuck at #3. Truth is, the library works blissfully for decoding, but the instructions on encoding back are very thin, to say the least. I've been reading through the code, trying to make sense of things, but to no success.

Data

Here's what I have:

res = {
  'extn_id': '2.5.29.35',
  'critical': False,
  'extn_value': {
    'key_identifier': b'\xf2\xc0\x13\xe0\x82C>\xfb\xee/g2\x965\\\xdb\xb8\xcb\x02\xd0',
    'authority_cert_issuer': [
      {
        'value_type': GeneralNameChoice.DIRECTORY_NAME,
        'value': {
          'rdn_sequence': [
            {
              'values': [
                {
                  'type': '2.5.4.6',
                  'value': 'BM',
                },
              ],
            },
            {
              'values': [
                {
                  'type': '2.5.4.10',
                  'value': 'QuoVadis Limited',
                },
              ],
            },
            {
              'values': [
                {
                  'type': '2.5.4.3',
                  'value': 'QuoVadis Root CA 3',
                },
              ],
            },
          ],
        },
      },
    ],
    'authority_cert_serial_number': 1478,
  },
}

The GeneralNameChoice.DIRECTORY_NAME is an enum that I use on this dictionary in order to get the appropriate alternative for GeneralName:

gn_value_types = {
    cert.GeneralNameChoice.OTHER_NAME: 'other_name',
    cert.GeneralNameChoice.RFC822_NAME: 'rfc822_name',
    cert.GeneralNameChoice.DNS_NAME: 'dns_name',
    cert.GeneralNameChoice.DIRECTORY_NAME: 'directory_name',
    cert.GeneralNameChoice.EDI_PARTY_NAME: 'edi_party_name',
    cert.GeneralNameChoice.UNIFORM_RESOURCE_IDENTIFIER: 'uniform_resource_identifier',
    cert.GeneralNameChoice.IP_ADDRESS: 'ip_address',
    cert.GeneralNameChoice.REGISTERED_ID: 'registered_id',
    cert.GeneralNameChoice.X400_ADDRESS: 'or_address',
}

which means that it will go with 'directory_name'.

Process

And here's what I'm doing:

res['extn_value']['authority_cert_issuer'] = list(map(
    lambda v: x509.GeneralNames([
        x509.GeneralName({
            gn_value_types[v['value_type']]: x509.Name.build(dict(
                [tuple(d['values'][0].values()) for d in v['value']['rdn_sequence']]
            ))
        })
    ]),
    res['extn_value']['authority_cert_issuer']
))

All that mess equals literally to:

res['extn_value']['authority_cert_issuer'] = [
  x509.GeneralNames([ x509.GeneralName({'directory_name': x509.Name.build({'country_name': 'BM', 'organization_name': 'QuoVadis Limited', 'common_name': 'QuoVadis Root CA 3'})}) ])
]

Analysis

Here's how the encoded GeneralNames look like:

# Original:
<asn1crypto.x509.GeneralNames 126041340246736 b'\xa1I\xa4G0E1\x0b0\t\x06\x03U\x04\x06\x13\x02BM1\x190\x17\x06\x03U\x04\n\x13\x10QuoVadis Limited1\x1b0\x19\x06\x03U\x04\x03\x13\x12QuoVadis Root CA 3'>
# Generated:
<asn1crypto.x509.GeneralNames 126041337843488 b'0I\xa4G0E1\x0b0\t\x06\x03U\x04\x06\x13\x02BM1\x190\x17\x06\x03U\x04\n\x0c\x10QuoVadis Limited1\x1b0\x19\x06\x03U\x04\x03\x0c\x12QuoVadis Root CA 3'>

It's immediately obvious that they differ somewhat. However, their native forms are identical, both like:

[OrderedDict({'country_name': 'BM', 'organization_name': 'QuoVadis Limited', 'common_name': 'QuoVadis Root CA 3'})]

and even their children:

[OrderedDict({'country_name': 'BM', 'organization_name': 'QuoVadis Limited', 'common_name': 'QuoVadis Root CA 3'})]

are identical:

>>> agn.children[0] == w.children[0]
True

where, agn is the GeneralNames decoded from file and w is the one generated.

Generating

Errors arise from generating via x509.Certificate(d), where d is a complex dictionary including the res extension above:

Exception: ValueError: Value [UNIVERSAL 16] did not match the class and tag of any of the alternatives in asn1crypto.x509.GeneralName: [CONTEXT 0], [CONTEXT 1], [CONTEXT 2], [CONTEXT 3], [CONTEXT 4], [CONTEXT 5], [CONTEXT 6], [CONTEXT 7], [CONTEXT 8]
    while constructing asn1crypto.x509.GeneralNames
    while constructing asn1crypto.x509.AuthorityKeyIdentifier
    while constructing asn1crypto.x509.Extension
    while constructing asn1crypto.x509.Extensions
    while constructing asn1crypto.x509.TbsCertificate
    while constructing asn1crypto.x509.Certificate

What I think is wrong

The fact that I have the .children identical suggests that the issue is not in x509.Name.build(...) or in x509.GeneralName(...), but rather in x509.GeneralNames(...). I think I might be missing some parameter.

Please let me know what do you think.

Alternate route

I tried your suggestion to issue #96, namely resorting to certbuilder instead, but I cannot make this work on my machine. It installs ok but as soon as I import it, I get errors:

oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto

Ny /usr/lib/x86_64-linux-gnu/libcrypto.so is a symlink to /usr/lib/x86_64-linux-gnu/libcrypto.so.3 , so I'm going to guess version 3. Could the error come from the fact that libcrypto.so is a symlink? Perhaps this is a question for another repo.

Conclusion

I wish there was a simpler way to build one such certificate from a native Python dictionary, such as the .native property of a certificate. Perhaps that's precisely what certbuilder is supposed to do, but I can't run it.

Any help would be very appreciated. It's my job, after all.

Answer 1 · 2024-08-23T08:24:05.000Z

Further to my previous question, and still under GeneralNames issues, I have succeeded in creating a GeneralNames with a GeneralName with a URI. However, when passing spaces in this, they get URL_encoded.

I checked asn1crypto.x509 and indeed, URI.set does self.contents = iri_to_uri(value).

What I find hard to understand — and impossible to replicate — is that when loading a certificate and inspecting its nodes, I get to the same GeneralName.chosen and this is an URI, but its contents have a clear space, whereas mine gets a %20. Naturally, this makes the generated certificate differ from the original one, and also makes me unsure how correct will my certificates be.

Furthermore, I added some prints to asn1crypto.x509 and to my surprise, although the loaded certificate does have URI nodes (with clear spaces!), I don't get any calls to URI.set.

For now, I'm doing this:

value = x509.URI(attribute_value)
value.contents = value.unicode # <============ OUCH!

Needless to say, this hack is not winning any beauty pageant. Please help. Please advise.

Answer 2 · 2024-08-23T09:40:29.000Z

Haven't looked at the code in detail, but from the binary dumps, this looks like a contextual tagging issue. I suspect the two encodings will be identical if you .untag() them. What happens if you do that?

Answer 3 · 2024-08-23T12:29:52.000Z

Hi @ricardo-reis-1970 , are you trying to encode the QuoVadis Root CA 3?

Your procedure (compare existing files to the ones you generate) is what I do too when working with ASN.1/DER.

Easiest way to decode+re-encode+compare is: Certificate.load(der).dump(True) == der

On my machine this is true for /etc/ssl/certs/QuoVadis_Root_CA_3.pem, i.e. it will be possible to create the same cert from scratch.

Another useful tool is dumpasn1.
For the extension I think you're looking at it prints:

  0 110: SEQUENCE {
  2   3: . OBJECT IDENTIFIER authorityKeyIdentifier (2 5 29 35)
  7 103: . OCTET STRING, encapsulates {
  9 101: . . SEQUENCE {
 11  20: . . . [0] F2 C0 13 E0 82 43 3E FB EE 2F 67 32 96 35 5C DB B8 CB 02 D0
 33  73: . . . [1] {
 35  71: . . . . [4] {
 37  69: . . . . . SEQUENCE {
 39  11: . . . . . . SET {
 41   9: . . . . . . . SEQUENCE {
 43   3: . . . . . . . . OBJECT IDENTIFIER countryName (2 5 4 6)
 48   2: . . . . . . . . PrintableString 'BM'
       : . . . . . . . . }
       : . . . . . . . }
 52  25: . . . . . . SET {
 54  23: . . . . . . . SEQUENCE {
 56   3: . . . . . . . . OBJECT IDENTIFIER organizationName (2 5 4 10)
 61  16: . . . . . . . . PrintableString 'QuoVadis Limited'
       : . . . . . . . . }
       : . . . . . . . }
 79  27: . . . . . . SET {
 81  25: . . . . . . . SEQUENCE {
 83   3: . . . . . . . . OBJECT IDENTIFIER commonName (2 5 4 3)
 88  18: . . . . . . . . PrintableString 'QuoVadis Root CA 3'
       : . . . . . . . . }
       : . . . . . . . }
       : . . . . . . }
       : . . . . . }
       : . . . . }
108   2: . . . [2] 05 C6
       : . . . }
       : . . }
       : . }

Code to create the original DER encoding of the extension:

#!/usr/bin/python3

from asn1crypto.x509 import Extension, GeneralName

orig_der = bytes.fromhex(
    "306e0603551d23046730658014f2c013e082433efbee2f673296355c"
    "dbb8cb02d0a149a4473045310b300906035504061302424d31193017"
    "060355040a131051756f5661646973204c696d69746564311b301906"
    "03550403131251756f566164697320526f6f742043412033820205c6"
)

ext = Extension({
    "extn_id": "authority_key_identifier",
    "extn_value": {
        "key_identifier": bytes.fromhex("F2 C0 13 E0 82 43 3E FB EE 2F 67 32 96 35 5C DB B8 CB 02 D0"),
        "authority_cert_issuer": [
            GeneralName({
                "directory_name": {"": [
                    [{"type": "country_name", "value": {"printable_string": "BM"}}],
                    [{"type": "organization_name", "value": {"printable_string": "QuoVadis Limited"}}],
                    [{"type": "common_name", "value": {"printable_string": "QuoVadis Root CA 3"}}],
                ]}
            })
        ],
        "authority_cert_serial_number": 0x05C6,
    },
})

print(orig_der == ext.dump())

Hope this helps.
If you're unsure how to generate a specific DER output, please provide the hex/base64/etc dump of the desired output and a complete+minimal testcase that's trying to generate it.

Answer 4 · 2024-08-23T12:45:45.000Z

What I find hard to understand — and impossible to replicate — is that when loading a certificate and inspecting its nodes, I get to the same GeneralName.chosen and this is an URI, but its contents have a clear space, whereas mine gets a %20. Naturally, this makes the generated certificate differ from the original one, and also makes me unsure how correct will my certificates be.

This is one of the situations where asn1crypto prevents you from creating incorrect certificates. I think RFC 3986 forbids using plain blanks in URLs.

Answer 5 · 2024-08-23T13:28:17.000Z

Hi @joernheissler.

Thank you for the tips. There's a lot there to try.

Let me start with the Certificate.load(der).dump(True) == der technique. Yes, this is a good way to sanity-check asn1crypto and I'm sure it's in the tests. However, this never leaves the asn1crypto scope. What I'm doing is trying to generate certificates from native Python values.

I need a reliable way to generate all aspects of a certificate in a way that they will end up like the original, but stepping out of the library scope. The goal is to later generate my own certificates, go home and sleep, knowing that I'm not messing them up.

Regarding the QuoVadis, I'm decoding / processing / reencoding these:

QuoVadis_Root_CA_1_G3.crt
QuoVadis_Root_CA_2.crt
QuoVadis_Root_CA_2_G3.crt
QuoVadis_Root_CA_3.crt
QuoVadis_Root_CA_3_G3.crt

as part of a larger set of 148, from my Ubuntu 24.04. I'm getting a 35% match between original and reference. These QuoVaids are all in the non-successful two thirds, I'm afraid.

But more specifically, I'm focusing on ICAO certificates, as those are the ones relevant for my job. In this case, I am trying to generate a GeneralNames for a value with a space:

x509.GeneralName({
    'uniform_resource_identifier': 'https://pkddownload1.icao.int/CRLs/SYC.crl https://pkddownload2.icao.int/CRLs/SYC.crl'
})

and the result comes back always as:

<asn1crypto.x509.GeneralName 128227210900544 b'\x86Whttps://pkddownload1.icao.int/CRLs/SYC.crl%20https://pkddownload2.icao.int/CRLs/SYC.crl'>

If I try more specifically:

>>> x509.URI('https://pkddownload1.icao.int/CRLs/SYC.crl https://pkddownload2.icao.int/CRLs/SYC.crl')
<asn1crypto.x509.URI 124578456528384 b'\x16Whttps://pkddownload1.icao.int/CRLs/SYC.crl%20https://pkddownload2.icao.int/CRLs/SYC.crl'>

I get the same.

This would not bother me at all, except that the original certificate does have a space instead of a %20. Here's the original, from ASN.1 JavaScript Decoder:

Extension SEQUENCE @1038+102 (constructed): (2 elem)
  extnID OBJECT_IDENTIFIER @1040+3: 2.5.29.31|cRLDistributionPoints|X.509 extension
  extnValue OCTET_STRING @1045+95 (encapsulates): (95 byte)|305D305BA059A057865568747470733A2F2F706B64646F776E6C6F6164312E6963616F2E696E742F43524C732F5359432E63726C2068747470733A2F2F706B64646F776E6C6F6164322E6963616F2E696E742F43524C732F5359432E63726C
    SEQUENCE @1047+93 (constructed): (1 elem)
      SEQUENCE @1049+91 (constructed): (1 elem)
        [0] @1051+89 (constructed): (1 elem)
          [0] @1053+87 (constructed): (1 elem)
            [6] @1055+85: (85 byte)|https://pkddownload1.icao.int/CRLs/SYC.crl https://pkddownload2.icao.int/CRLs/SYC.crl

And here's mine:

Extension SEQUENCE @1038+104 (constructed): (2 elem)
  extnID OBJECT_IDENTIFIER @1040+3: 2.5.29.31|cRLDistributionPoints|X.509 extension
  extnValue OCTET_STRING @1045+97 (encapsulates): (97 byte)|305F305DA05BA059865768747470733A2F2F706B64646F776E6C6F6164312E6963616F2E696E742F43524C732F5359432E63726C25323068747470733A2F2F706B64646F776E6C6F6164322E6963616F2E696E742F43524C732F5359432E63726C
    SEQUENCE @1047+95 (constructed): (1 elem)
      SEQUENCE @1049+93 (constructed): (1 elem)
        [0] @1051+91 (constructed): (1 elem)
          [0] @1053+89 (constructed): (1 elem)
            [6] @1055+87: (87 byte)|https://pkddownload1.icao.int/CRLs/SYC.crl%20https://pkddownload2.icao.int/CRLs/SYC.crl

The sequence is 2 bytes longer (length diff between " " and "%20").

As I said, asn1crypto.x509.URI.set has a clear line where the contents are url_encoded, but this set method is not called when loading this ICAO certificate, or any certificate (I checked).

I do get lost while following the whole class hierarchy and determining exactly which __init__ does what.

My attempt to cheat by changing the contents property is obviously breaking because the length gets shorter (85), but it still expects to read 87 bytes. I could not find a length property that I could also cheat into turning 85. In any case, I don't find these hacks any good.

I'm attaching the culprit file. The issue comes up in the last extension: cRLDistributionPoints.

csca.zip

Answer 6 · 2024-08-23T13:31:58.000Z

I came across Peter Gutmann's dumpasn1 a couple days too late. I had already developed mine in C, which was great fun. In any case, I reached out to Tom and he replied very promptly (!), urging me to not be inspired by that tool. Brilliant, and modest.

The tool is great.

Answer 7 · 2024-08-23T14:54:45.000Z

The issue comes up in the last extension: cRLDistributionPoints.

I assume that ICAO generated an incorrect certificate by using space separated URLs, but X.509 and related standards may be very subtle, so perhaps it's valid after all.

Also, the "CRL Distribution Points" is usually not put into the root certificate, but into the issued certificates. Again, I might be missing something here.

Answer 8 · 2024-08-23T14:58:01.000Z

Let me start with the Certificate.load(der).dump(True) == der technique. Yes, this is a good way to sanity-check asn1crypto and I'm sure it's in the tests. However, this never leaves the asn1crypto scope. What I'm doing is trying to generate certificates from native Python values.

I need a reliable way to generate all aspects of a certificate in a way that they will end up like the original, but stepping out of the library scope. The goal is to later generate my own certificates, go home and sleep, knowing that I'm not messing them up.

I gave an example how to create a single Extension manually. Same can be done for a whole certificate.

Answer 9 · 2024-10-15T08:46:01.000Z

Dear all,

I want to thank all contributions. As a newcomer, this has been a very instructive journey.

@joernheissler you are 100% correct: ICAO generates incorrect certificates. In the CRLDistributionPoints extension they just squeeze two URLs together in one DistributionPoint, instead of adding two separate DistributionPoints. I'm not sure how this happened, but I guess it's safe to say they are not using asn1crypto.

The offending certificates have been failing other sanity checks, so I'm using them now as failure markers for tests.

I'm closing this issue.