Older certs failing on CER/DER/BER confusion for Boolean value.

[GoogleCodeExporter]

I ran the cert pasing on the some random certs live in the wild.

These work "fine" in browsers, and openssl (although cert might expired or dead 
or self-signed or not trusted)

Many are failing due to this check deep inside:

pyasn1/codec/cer/decoder.py

class BooleanDecoder(decoder.AbstractSimpleDecoder):
    protoComponent = univ.Boolean(0)
    def valueDecoder(self, fullSubstrate, substrate, asn1Spec, tagSet, length,
                     state, decodeFun, substrateFun):
        head, tail = substrate[:length], substrate[length:]
        if not head:
            raise error.PyAsn1Error('Empty substrate')
        byte = oct2int(head[0])
        # CER/DER specifies encoding of TRUE as 0xFF and FALSE as 0x0, while
        # BER allows any non-zero value as TRUE; cf. sections 8.2.2. and 11.1
        # in http://www.itu.int/ITU-T/studygroups/com17/languages/X.690-0207.pdf
        if byte == 0xff:
            value = 1
        elif byte == 0x00:
            value = 0
        else:
            raise error.PyAsn1Error('Boolean CER violation: %s' % byte)

(with value of "1")

I don't know the field exactly (the backtrace is a nightmare).  However, I can 
confirm that changing the logic to not raise the exception (and setting 
value=1) allows it to continue.

I believe we can create a decoder that allows this type of boolean value.

Or we can keep process as is, and just raise an somewhat cryptic exception.

your thoughts?

nickg


here's a sample.. this is live on the internets...

-----BEGIN CERTIFICATE-----
MIICgjCCAeugAwIBAgIBADANBgkqhkiG9w0BAQUFADB8MSMwIQYDVQQDDBpNYXRy
aXhTU0wgU2FtcGxlIFNlcnZlciBDQTELMAkGA1UEBgwCVVMxCzAJBgNVBAgMAldB
MREwDwYDVQQHDAhCZWxsZXZ1ZTEZMBcGA1UECgwQUGVlcnNlYyBOZXR3b3JrczEN
MAsGA1UECwwEVGVzdDAeFw0wNjAzMTMwODEzMzRaFw0wNzAzMTMwODEzMzRaMH4x
JTAjBgNVBAMMHE1hdHJpeFNTTCBTYW1wbGUgU2VydmVyIENlcnQxCzAJBgNVBAYM
AlVTMQswCQYDVQQIDAJXQTERMA8GA1UEBwwIQmVsbGV2dWUxGTAXBgNVBAoMEFBl
ZXJTZWMgTmV0d29ya3MxDTALBgNVBAsMBFRlc3QwgZ4wDQYJKoZIhvcNAQEBBQAD
gYwAMIGIAoGArLJFx36VVxymSgNnepNJfNQ2Wh6RUoe3n6YakhtPWCXHRvGwNBr6
lOqt5bZkcsqM2FkUYMAdIV3DTvMXAVOUf9q0Ys27oIDny/0h2D8vVehjIScYrMrm
xPULuVuovFW4q+VMu+5vMp7U29j0zOHfn/xe687X7DjyaMd19eCcHiECAwEAAaMT
MBEwDwYDVR0TAQEBBAUwAwEBADANBgkqhkiG9w0BAQUFAAOBgQBX6R0aLW/agkug
2zl7kOxssCzi2l55Co6oGno+9t4Xk2IMVpIU/gas8RwnQRjFhw1M02yWobxYEh32
WRJdomY7VNB3m73VhMX909fP+m/d4shJNwB35oZn8JZPjmk9NB/ZGvxK6Zk0WttV
F/KVvu5//R80bStnBA4cAhP7FyUugg==
-----END CERTIFICATE-----

Original issue reported on code.google.com by nickgsup...@gmail.com on 1 Oct 2013 at 12:18

[GoogleCodeExporter]

This actually seems fairly common.  I'm in the midst of parsing 1,800,000  
SSL/TLS chains (so x2 x3 that many certificates).

I have a patch (it's pretty small too), but I'll see the % of certs with this 
issue first.

Original comment by nickgsup...@gmail.com on 1 Oct 2013 at 2:33

[GoogleCodeExporter]

Yikes. (But a very helpful report, thanks a lot!)

My longer-term plan is to replace pyasn1 with a custom decoder (mostly for 
performance reasons).

Using pyasn1.codec.ber instead of pyasn1.codec.der should be an OK short-term 
fix (since the only DER rule that pyasn1 appears to enforce in the "DER" 
decoder is the Boolean rule anyway). I'll commit the fix once I've confirmed 
that this doesn't break more than it fixes.

Original comment by ekasper@google.com on 1 Oct 2013 at 3:32

• Changed state: Accepted

[GoogleCodeExporter]

My scan is still running but the  short answer is there appears to be some 
certs issued by http://www.matrixssl.org for embedded systems that have this 
formatting bug.  This explains why some are very old -- they are embedded 
systems and  are never getting upgraded.

Approximately 0.75%  (thats < 1% not 75%) of all IPs that use SSL have this 
cert.  IMHO that's big enough to want to be able to parse these correctly OR 
just hardwire this cert's info in.

cert.py
+from ct.crypto.asn1 import derdecoder as der_decoder
+#from pyasn1.codec.der import decoder as der_decoder

new file:
"""
A DER decoder with various "adjustments" to parse wild certificates
"""

from pyasn1.type import univ
from pyasn1.codec.cer import decoder
from pyasn1.codec.ber.decoder import BooleanDecoder

tagMap = decoder.tagMap.copy()
tagMap.update({
    univ.Boolean.tagSet: BooleanDecoder()
    })
typeMap = decoder.typeMap
Decoder = decoder.Decoder

decode = Decoder(tagMap, typeMap)

I'll make real patch.

nickg

Original comment by nickgsup...@gmail.com on 1 Oct 2013 at 3:33

[GoogleCodeExporter]

ahh we crossed paths...

good idea on using custom ASN1.

using the patch above seems to work, but your trick of

from pyasn1.codec.BER import decoder as der_decoder

is more clever!

Original comment by nickgsup...@gmail.com on 1 Oct 2013 at 3:36

[GoogleCodeExporter]

over-night, I parsed 966026 live certs.

Of those 6313 has 'Invalid DER encoding: Boolean CER violation' problem

Of those 18 are some other cert, and the remaining is the default MatrixSSL 
cert.

>>> (6313 / 966026.0) * 100.0
0.6535020796541708 %

These appears to be home WiMAX routers.  nice.

Original comment by nickgsup...@gmail.com on 1 Oct 2013 at 11:34

[GoogleCodeExporter]

https://codereview.appspot.com/14270043

1 line change + tests + sample cert

Original comment by nickgsup...@gmail.com on 2 Oct 2013 at 9:44

[GoogleCodeExporter]

Temporary fix committed in
https://code.google.com/p/certificate-transparency/source/detail?r=1945a4e4e00d

Original comment by ekasper@google.com on 2 Oct 2013 at 1:15

• Changed state: Fixed