OCTWRAP, SEQWRAP, SETWRAP, BITWRAP
Closed this issue · 13 comments
EDIT: the cause of this is ambiguity over an old feature in OpenSSL's version of ascii2der, the BITWRAP type, which, much like a void* in C, breaks out of the ASN.1 type system for times that you need that.
I made myself a secp256r1 key with openssl and converted it to a public key (btw OpenSSL's prime256v1 == NIST's P-256 == secp256r1):
$ openssl ecparam -name prime256v1 -genkey -noout -out icantbelieveitsnotyubikey.pem
$ openssl ec -in icantbelieveitsnotyubikey.pem -pubout -outform DER -out icantbelieveitsnotyubikey.pub.der
And I got these files (mind, they aren't actually .txts, github just sensibly has a file extension filter on):
icantbelieveitsnotyubikey.pub.der
OpenSSL parses each like this:
$ openssl ec -in icantbelieveitsnotyubikey.pem -text
read EC key
Private-Key: (256 bit)
priv:
00:82:b3:19:91:10:37:a3:ea:21:76:99:66:c4:28:
de:ba:ec:be:e7:9a:c4:9f:87:29:d0:c9:1c:c7:5d:
e1:e6:4e
pub:
04:3f:d3:8b:c5:df:75:d6:ca:e0:55:cf:81:60:b2:
32:77:d2:95:f5:9a:8e:34:a2:b0:dc:69:b8:1a:f9:
0a:1c:04:2c:02:c9:55:80:e6:9e:88:37:e7:52:ec:
a1:39:5e:30:a9:20:22:83:ec:1a:85:f8:da:a7:8a:
de:83:3c:75:2b
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIIKzGZEQN6PqIXaZZsQo3rrsvueaxJ+HKdDJHMdd4eZOoAoGCCqGSM49
AwEHoUQDQgAEP9OLxd911srgVc+BYLIyd9KV9ZqONKKw3Gm4GvkKHAQsAslVgOae
iDfnUuyhOV4wqSAig+wahfjap4regzx1Kw==
-----END EC PRIVATE KEY-----
$ openssl asn1parse -inform DER -in icantbelieveitsnotyubikey.pub.der -dump
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 3f d3 8b c5 df 75-d6 ca e0 55 cf 81 60 b2 ..?....u...U..`.
0010 - 32 77 d2 95 f5 9a 8e 34-a2 b0 dc 69 b8 1a f9 0a 2w.....4...i....
0020 - 1c 04 2c 02 c9 55 80 e6-9e 88 37 e7 52 ec a1 39 ..,..U....7.R..9
0030 - 5e 30 a9 20 22 83 ec 1a-85 f8 da a7 8a de 83 3c ^0. "..........<
0040 - 75 2b u+ ..
That all makes sense to me: openssl ec -text's "pub:" output matches the BITSTRING object in the DER (ignoring the \x00 header), and that object has 65 bytes starting with a \x04 so it's a secg uncompressed elliptic curve point.
But der2ascii crops the first two bytes off the pubkey:
$ der2ascii -i icantbelieveitsnotyubikey.pub.der.txt
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `d38bc5df75d6cae055cf8160b23277d295f59a8e34a2b0dc69b81af90a1c042c02c95580e69e8837e752eca1395e30a9202283ec1a85f8daa78ade833c752b` }
}
}
Notice that the tail is correct, (de 83 3c 75 2b), and if you check the rest it's all there, and the leading \x00 is too, all except for the first \x04 secg format header, and the first byte of the first coordinate.
I have macOS 10.14.4,
$ openssl version # installed from brew
OpenSSL 1.0.2r 26 Feb 2019
$ go version
go version go1.12.4 darwin/amd64
and the latest der2ascii from go get github.com/google/der-ascii/cmd/....
If I try to reproduce by rolling another key, I can't. I must have hit a very unlucky case that somehow trips your parser.
$ pushd `mktemp -d`
/var/folders/sy/qyp_73v51_757c_h339_h9xw0000gn/T/tmp.cx5fFJVq ~/
$ openssl ecparam -name prime256v1 -genkey -noout -out icantbelieveitsnotyubikey.pem
$ openssl ec -in icantbelieveitsnotyubikey.pem -text
read EC key
Private-Key: (256 bit)
priv:
08:c7:9f:61:c3:c4:df:c5:8e:fb:6d:87:29:e3:70:
3a:63:22:4d:ae:e1:6e:e4:52:59:c1:c6:fc:53:35:
0c:e5
pub:
04:09:0a:62:a9:da:4e:e0:44:61:d6:6c:0c:12:88:
16:5e:4a:54:fb:fe:6d:9d:e1:bb:79:7c:fe:d7:93:
f1:f5:b2:94:f6:68:02:9a:08:0e:80:0a:07:19:1d:
d9:04:76:d1:ed:8d:53:2e:cc:79:76:6d:ba:2b:1b:
7a:bc:fc:56:59
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIAjHn2HDxN/FjvtthynjcDpjIk2u4W7kUlnBxvxTNQzloAoGCCqGSM49
AwEHoUQDQgAECQpiqdpO4ERh1mwMEogWXkpU+/5tneG7eXz+15Px9bKU9mgCmggO
gAoHGR3ZBHbR7Y1TLsx5dm26Kxt6vPxWWQ==
-----END EC PRIVATE KEY-----
$ openssl ec -in icantbelieveitsnotyubikey.pem -pubout -outform DER -out icantbelieveitsnotyubikey.pub.der
read EC key
writing EC key
$ openssl asn1parse -inform DER -in icantbelieveitsnotyubikey.pub.der -dump
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 09 0a 62 a9 da 4e-e0 44 61 d6 6c 0c 12 88 ....b..N.Da.l...
0010 - 16 5e 4a 54 fb fe 6d 9d-e1 bb 79 7c fe d7 93 f1 .^JT..m...y|....
0020 - f5 b2 94 f6 68 02 9a 08-0e 80 0a 07 19 1d d9 04 ....h...........
0030 - 76 d1 ed 8d 53 2e cc 79-76 6d ba 2b 1b 7a bc fc v...S..yvm.+.z..
0040 - 56 59 VY
$ der2ascii -i icantbelieveitsnotyubikey.pub.der
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING { `00` `04090a62a9da4ee04461d66c0c1288165e4a54fbfe6d9de1bb797cfed793f1f5b294f668029a080e800a07191dd90476d1ed8d532ecc79766dba2b1b7abcfc5659` }
}
^ This bit_string is correctly 65 bytes long, and starts with 0409 just like asn1parse thinks it should.
I believe what happened here is you got unlucky. :-( der2ascii's output is a valid decode of that input (you can check this by running ascii2der and comparing), but not the useful one. About 1 in 256 EC keys will have this problem.
The first byte of your public key, excluding the 0x04 uncompressed marker, happens to be 3f, which is 63. A DER element is tag, length, value, where tags are (usually) single-byte, as are lengths under 128. 0x04 is the tag for OCTET STRING, and a byte length of 63 does indeed consume the entirety of the remaining BIT STRING. So der2ascii guesses that you've embedded an ASN.1 structure inside a BIT STRING and decodes a layer down. It does this because SubjectPublicKeyInfos (that structure you're decoding here) sometimes embed DER inside BIT STRING, notably with RSA keys, as do X.509 signatures (notably with ECDSA).
I'm not sure what can be done about this one. I can add a hack to say that a BIT STRING containing an OCTET STRING doesn't recurse, but I'm pretty sure some X.509 extensions do have OCTET STRING payloads. (Though those use OCTET STRING on the outside rather than BIT STRING.) I think an early iteration of Ed25519's embedding in X.509 would have triggered just this case, though they were since convinced to not waste those two bytes.
I wrote a script to fuzz der2ascii and found more examples:
while true; do
until [ $(openssl ecparam -name prime256v1 -genkey -noout -out icantbelieveitsnotyubikey.pem &&
openssl ec -in icantbelieveitsnotyubikey.pem -pubout -outform DER |
der2ascii |
grep "BIT_STRING" |
awk '{print $4}' |
wc -c) -ne 133 ]; do # 133 is 65 bytes **in hex** + two quotes + newline
echo -n;
done
openssl ec -in icantbelieveitsnotyubikey.pem -pubout -outform DER -out icantbelieveitsnotyubikey.pub.der
openssl ec -in icantbelieveitsnotyubikey.pem -text
openssl asn1parse -inform DER -in icantbelieveitsnotyubikey.pub.der -dump
der2ascii -i icantbelieveitsnotyubikey.pub.der
ln -f icantbelieveitsnotyubikey.pem icantbelieveitsnotyubikey.pem.txt
ln -f icantbelieveitsnotyubikey.pub.der icantbelieveitsnotyubikey.pub.der.txt
read -p "Upload fail keys and press enter to find another."
done
There are three outputs per example, two from openssl, showing the correct output, and the last from der2ascii, which differs.
icantbelieveitsnotyubikey.pem.txt
icantbelieveitsnotyubikey.pub.der.txt
read EC key
Private-Key: (256 bit)
priv:
0c:7b:37:26:95:53:6c:b3:2b:cb:44:c9:5d:4c:68:
d5:f1:27:26:09:4e:a2:41:e2:68:9e:64:b1:13:25:
53:4f
pub:
04:3f:6e:77:b8:b8:00:7f:6c:1e:1d:e6:39:a9:a0:
5f:79:03:d4:4b:1e:e1:9c:d7:0a:15:7b:64:5d:9f:
5d:da:97:da:c1:79:e0:89:17:0f:74:67:d6:a8:a0:
bb:75:85:7d:e4:04:36:a0:10:92:31:6b:01:0e:c0:
22:4e:37:72:29
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIAx7NyaVU2yzK8tEyV1MaNXxJyYJTqJB4mieZLETJVNPoAoGCCqGSM49
AwEHoUQDQgAEP253uLgAf2weHeY5qaBfeQPUSx7hnNcKFXtkXZ9d2pfawXngiRcP
dGfWqKC7dYV95AQ2oBCSMWsBDsAiTjdyKQ==
-----END EC PRIVATE KEY-----
$ openssl asn1parse -inform DER -in icantbelieveitsnotyubikey.pub.der -dump
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 3f 6e 77 b8 b8 00-7f 6c 1e 1d e6 39 a9 a0 ..?nw....l...9..
0010 - 5f 79 03 d4 4b 1e e1 9c-d7 0a 15 7b 64 5d 9f 5d _y..K......{d].]
0020 - da 97 da c1 79 e0 89 17-0f 74 67 d6 a8 a0 bb 75 ....y....tg....u
0030 - 85 7d e4 04 36 a0 10 92-31 6b 01 0e c0 22 4e 37 .}..6...1k..."N7
0040 - 72 29 r)
$ der2ascii -i icantbelieveitsnotyubikey.pub.der
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `6e77b8b8007f6c1e1de639a9a05f7903d44b1ee19cd70a157b645d9f5dda97dac179e089170f7467d6a8a0bb75857de40436a01092316b010ec0224e377229` }
}
}
icantbelieveitsnotyubikey.pem.txt
icantbelieveitsnotyubikey.pub.der.txt
Private-Key: (256 bit)
priv:
00:bf:47:85:c1:89:ed:13:4d:fa:16:bc:4e:02:90:
19:69:07:c6:9f:2c:ae:f6:46:88:47:61:93:df:33:
4c:aa:94
pub:
04:26:4a:6f:38:87:22:8e:4c:b8:67:c7:97:b3:e3:
d7:5b:00:c8:f2:8e:db:72:4d:3f:ab:f1:f6:66:81:
1f:e3:a1:d9:a7:79:04:e5:9f:32:d3:17:54:56:46:
84:a9:f5:54:a2:e8:44:b5:e6:e0:30:19:a6:e5:77:
8c:d5:27:09:a7
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIL9HhcGJ7RNN+ha8TgKQGWkHxp8srvZGiEdhk98zTKqUoAoGCCqGSM49
AwEHoUQDQgAEJkpvOIcijky4Z8eXs+PXWwDI8o7bck0/q/H2ZoEf46HZp3kE5Z8y
0xdUVkaEqfVUouhEtebgMBmm5XeM1ScJpw==
-----END EC PRIVATE KEY-----
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 26 4a 6f 38 87 22-8e 4c b8 67 c7 97 b3 e3 ..&Jo8.".L.g....
0010 - d7 5b 00 c8 f2 8e db 72-4d 3f ab f1 f6 66 81 1f .[.....rM?...f..
0020 - e3 a1 d9 a7 79 04 e5 9f-32 d3 17 54 56 46 84 a9 ....y...2..TVF..
0030 - f5 54 a2 e8 44 b5 e6 e0-30 19 a6 e5 77 8c d5 27 .T..D...0...w..'
0040 - 09 a7 ..
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `4a6f3887228e4cb867c797b3e3d75b00c8f28edb724d3fabf1f666811fe3a1d9a77904e59f32` }
[PRIVATE 19 PRIMITIVE] { `54564684a9f554a2e844b5e6e03019a6e5778cd52709a7` }
}
}
icantbelieveitsnotyubikey.pem.txt
icantbelieveitsnotyubikey.pub.der.txt
read EC key
Private-Key: (256 bit)
priv:
00:88:ff:d4:90:9e:d5:c0:7a:14:1e:f9:ea:25:5f:
e4:14:f9:0a:6d:db:da:f3:67:08:9a:fc:f8:53:02:
e2:1f:80
pub:
04:3f:6c:71:a2:2e:cb:73:04:4c:6a:c5:83:b3:3f:
3a:89:46:a6:f8:a1:1d:ee:ec:4f:d3:d0:9d:7e:72:
0e:7b:0b:fc:e7:89:89:a7:c2:bc:d9:45:f3:ea:a2:
70:b7:d5:98:f7:49:8b:f5:af:90:c3:fc:43:ef:24:
a8:c0:bd:34:cf
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIIj/1JCe1cB6FB756iVf5BT5Cm3b2vNnCJr8+FMC4h+AoAoGCCqGSM49
AwEHoUQDQgAEP2xxoi7LcwRMasWDsz86iUam+KEd7uxP09CdfnIOewv854mJp8K8
2UXz6qJwt9WY90mL9a+Qw/xD7ySowL00zw==
-----END EC PRIVATE KEY-----
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 3f 6c 71 a2 2e cb-73 04 4c 6a c5 83 b3 3f ..?lq...s.Lj...?
0010 - 3a 89 46 a6 f8 a1 1d ee-ec 4f d3 d0 9d 7e 72 0e :.F......O...~r.
0020 - 7b 0b fc e7 89 89 a7 c2-bc d9 45 f3 ea a2 70 b7 {.........E...p.
0030 - d5 98 f7 49 8b f5 af 90-c3 fc 43 ef 24 a8 c0 bd ...I......C.$...
0040 - 34 cf 4.
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `6c71a22ecb73044c6ac583b33f3a8946a6f8a11deeec4fd3d09d7e720e7b0bfce78989a7c2bcd945f3eaa270b7d598f7498bf5af90c3fc43ef24a8c0bd34cf` }
}
}
icantbelieveitsnotyubikey.pem.txt
icantbelieveitsnotyubikey.pub.der.txt
read EC key
Private-Key: (256 bit)
priv:
00:ac:6c:8a:4c:f5:f9:31:45:81:c9:89:05:3b:fd:
44:75:80:f5:57:1d:c5:41:31:84:71:b6:9e:4e:ec:
ad:ee:10
pub:
04:3f:c4:a0:ab:aa:9b:ad:ba:0d:87:d8:0b:4d:c4:
59:87:cd:e1:3b:71:36:4f:78:7a:cb:70:86:9d:b6:
0b:97:cc:fd:59:08:87:b8:8a:19:58:67:40:3c:13:
93:10:75:5a:34:f2:e7:97:4d:a9:ea:5b:59:9e:92:
40:1c:92:1d:e6
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIKxsikz1+TFFgcmJBTv9RHWA9VcdxUExhHG2nk7sre4QoAoGCCqGSM49
AwEHoUQDQgAEP8Sgq6qbrboNh9gLTcRZh83hO3E2T3h6y3CGnbYLl8z9WQiHuIoZ
WGdAPBOTEHVaNPLnl02p6ltZnpJAHJId5g==
-----END EC PRIVATE KEY-----
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 3f c4 a0 ab aa 9b-ad ba 0d 87 d8 0b 4d c4 ..?...........M.
0010 - 59 87 cd e1 3b 71 36 4f-78 7a cb 70 86 9d b6 0b Y...;q6Oxz.p....
0020 - 97 cc fd 59 08 87 b8 8a-19 58 67 40 3c 13 93 10 ...Y.....Xg@<...
0030 - 75 5a 34 f2 e7 97 4d a9-ea 5b 59 9e 92 40 1c 92 uZ4...M..[Y..@..
0040 - 1d e6 ..
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `c4a0abaa9badba0d87d80b4dc45987cde13b71364f787acb70869db60b97ccfd590887b88a195867403c139310755a34f2e7974da9ea5b599e92401c921de6` }
}
}
icantbelieveitsnotyubikey.pem.txt
icantbelieveitsnotyubikey.pub.der.txt
read EC key
Private-Key: (256 bit)
priv:
42:d9:63:99:30:64:fc:58:ee:de:1d:9f:ec:21:17:
f9:ae:ba:59:ae:35:16:f6:2d:b9:1b:07:45:6f:10:
49:6f
pub:
04:3f:da:9d:02:7b:1b:c6:36:65:7a:1a:76:fb:3b:
60:32:1e:38:97:0a:4a:36:8a:cf:ed:c5:89:e8:f5:
b4:ee:f0:bb:8d:0d:81:ee:3f:ad:48:d1:c3:09:53:
d4:5f:39:72:56:de:ce:28:e8:d2:a2:81:91:8c:c6:
e3:dd:30:dc:c6
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIELZY5kwZPxY7t4dn+whF/muulmuNRb2LbkbB0VvEElvoAoGCCqGSM49
AwEHoUQDQgAEP9qdAnsbxjZlehp2+ztgMh44lwpKNorP7cWJ6PW07vC7jQ2B7j+t
SNHDCVPUXzlyVt7OKOjSooGRjMbj3TDcxg==
-----END EC PRIVATE KEY-----
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 3f da 9d 02 7b 1b-c6 36 65 7a 1a 76 fb 3b ..?...{..6ez.v.;
0010 - 60 32 1e 38 97 0a 4a 36-8a cf ed c5 89 e8 f5 b4 `2.8..J6........
0020 - ee f0 bb 8d 0d 81 ee 3f-ad 48 d1 c3 09 53 d4 5f .......?.H...S._
0030 - 39 72 56 de ce 28 e8 d2-a2 81 91 8c c6 e3 dd 30 9rV..(.........0
0040 - dc c6 ..
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `da9d027b1bc636657a1a76fb3b60321e38970a4a368acfedc589e8f5b4eef0bb8d0d81ee3fad48d1c30953d45f397256dece28e8d2a281918cc6e3dd30dcc6` }
}
}
icantbelieveitsnotyubikey.pem.txt
icantbelieveitsnotyubikey.pub.der.txt
read EC key
Private-Key: (256 bit)
priv:
00:93:0d:91:a5:dc:79:e9:53:01:4a:4d:03:d3:01:
75:22:e1:6a:16:8d:b4:3d:11:32:07:5a:0d:c5:d3:
cf:c8:65
pub:
04:3f:8e:a8:cf:5c:3a:ac:f0:4a:0a:dc:b9:aa:a9:
7e:74:a6:44:f6:4f:5c:79:f7:1f:a8:1c:02:94:2b:
a1:fc:69:c2:78:14:6d:99:f6:ed:15:9a:bc:c5:b4:
9f:96:f2:05:57:ab:58:16:29:c3:37:86:4d:94:bb:
c2:15:71:3c:14
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIJMNkaXceelTAUpNA9MBdSLhahaNtD0RMgdaDcXTz8hloAoGCCqGSM49
AwEHoUQDQgAEP46oz1w6rPBKCty5qql+dKZE9k9cefcfqBwClCuh/GnCeBRtmfbt
FZq8xbSflvIFV6tYFinDN4ZNlLvCFXE8FA==
-----END EC PRIVATE KEY-----
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 3f 8e a8 cf 5c 3a-ac f0 4a 0a dc b9 aa a9 ..?...\:..J.....
0010 - 7e 74 a6 44 f6 4f 5c 79-f7 1f a8 1c 02 94 2b a1 ~t.D.O\y......+.
0020 - fc 69 c2 78 14 6d 99 f6-ed 15 9a bc c5 b4 9f 96 .i.x.m..........
0030 - f2 05 57 ab 58 16 29 c3-37 86 4d 94 bb c2 15 71 ..W.X.).7.M....q
0040 - 3c 14 <.
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `8ea8cf5c3aacf04a0adcb9aaa97e74a644f64f5c79f71fa81c02942ba1fc69c278146d99f6ed159abcc5b49f96f20557ab581629c337864d94bbc215713c14` }
}
}
icantbelieveitsnotyubikey.pem.txt
icantbelieveitsnotyubikey.pub.der.txt
read EC key
Private-Key: (256 bit)
priv:
45:82:6f:21:be:fb:32:3b:ac:3f:c3:53:e6:7a:c5:
57:ed:7d:ff:a1:4a:c5:07:00:1b:70:f5:55:82:5f:
18:e2
pub:
04:3f:4b:93:ca:69:1f:57:97:b8:d9:2a:c5:cb:73:
ba:c0:a9:6e:49:8c:57:1a:d9:59:e1:6c:2f:e2:a4:
b7:04:5c:69:03:c5:45:2c:86:3f:26:b4:1a:26:62:
ab:75:ee:cc:e4:3d:d3:44:d6:77:91:75:dd:f9:4c:
3b:c6:f0:0b:04
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIEWCbyG++zI7rD/DU+Z6xVftff+hSsUHABtw9VWCXxjioAoGCCqGSM49
AwEHoUQDQgAEP0uTymkfV5e42SrFy3O6wKluSYxXGtlZ4Wwv4qS3BFxpA8VFLIY/
JrQaJmKrde7M5D3TRNZ3kXXd+Uw7xvALBA==
-----END EC PRIVATE KEY-----
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 3f 4b 93 ca 69 1f-57 97 b8 d9 2a c5 cb 73 ..?K..i.W...*..s
0010 - ba c0 a9 6e 49 8c 57 1a-d9 59 e1 6c 2f e2 a4 b7 ...nI.W..Y.l/...
0020 - 04 5c 69 03 c5 45 2c 86-3f 26 b4 1a 26 62 ab 75 .\i..E,.?&..&b.u
0030 - ee cc e4 3d d3 44 d6 77-91 75 dd f9 4c 3b c6 f0 ...=.D.w.u..L;..
0040 - 0b 04 ..
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `4b93ca691f5797b8d92ac5cb73bac0a96e498c571ad959e16c2fe2a4b7045c6903c5452c863f26b41a2662ab75eecce43dd344d6779175ddf94c3bc6f00b04` }
}
}
icantbelieveitsnotyubikey.pem.txt
icantbelieveitsnotyubikey.pub.der.txt
read EC key
Private-Key: (256 bit)
priv:
00:ee:58:e1:9d:b6:f2:bc:5d:05:42:40:47:e6:04:
ee:f3:cb:98:ca:b4:56:4e:e9:ad:1b:4e:f9:50:21:
2c:98:6a
pub:
04:3f:63:56:fb:b4:44:3d:ad:26:4a:63:65:c7:a9:
54:26:16:dc:a8:84:a7:25:44:d0:70:32:c0:8a:2b:
ed:70:d2:b5:20:e1:72:12:68:d2:37:25:d7:ed:3b:
f5:f4:e4:a4:17:de:7c:08:c1:5c:08:fc:6d:8b:70:
0c:33:4b:52:06
ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIO5Y4Z228rxdBUJAR+YE7vPLmMq0Vk7prRtO+VAhLJhqoAoGCCqGSM49
AwEHoUQDQgAEP2NW+7REPa0mSmNlx6lUJhbcqISnJUTQcDLAiivtcNK1IOFyEmjS
NyXX7Tv19OSkF958CMFcCPxti3AMM0tSBg==
-----END EC PRIVATE KEY-----
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 19 cons: SEQUENCE
4:d=2 hl=2 l= 7 prim: OBJECT :id-ecPublicKey
13:d=2 hl=2 l= 8 prim: OBJECT :prime256v1
23:d=1 hl=2 l= 66 prim: BIT STRING
0000 - 00 04 3f 63 56 fb b4 44-3d ad 26 4a 63 65 c7 a9 ..?cV..D=.&Jce..
0010 - 54 26 16 dc a8 84 a7 25-44 d0 70 32 c0 8a 2b ed T&.....%D.p2..+.
0020 - 70 d2 b5 20 e1 72 12 68-d2 37 25 d7 ed 3b f5 f4 p.. .r.h.7%..;..
0030 - e4 a4 17 de 7c 08 c1 5c-08 fc 6d 8b 70 0c 33 4b ....|..\..m.p.3K
0040 - 52 06 R.
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
OCTET_STRING { `6356fbb4443dad264a6365c7a9542616dca884a72544d07032c08a2bed70d2b520e1721268d23725d7ed3bf5f4e4a417de7c08c15c08fc6d8b700c334b5206` }
}
}
Oo I see. Thanks for the analysis, @davidben.
I don't know what to do about it either, but I'm glad you figured it out so quick!
And the second example in my post above, "00 04 26 4a 6f 38 87 ...", that decodes to "[PRIVATE 19 PRIMITIVE]", that's doing the same thing, but in that case it decodes 0x26 = 38 bytes as a string and then discovers bytes following that that happen to decode to a legitimate ASN.1 struct? Is der2ascii trying to decode an embedded structure and backtracking if it can't?
I wonder what OpenSSL is doing in this case. Maybe they special case ECDSA keys? :/
OpenSSL is just not recursing into BIT STRINGs and OCTET STRINGs like this. It means it'll never guess wrong like this, but it also won't decode embedded structures like this, which is less useful if you want to, say, concoct a certificate with a negative RSA key to see how different stacks behave. (That was one of my original use cases for the tool.)
I appreciate the quick responses. I see the bind you're in!
I've done some more exploring and I'd like to share my discoveries.
I learned yesterday that openssl asn1parse has a -strparse for handling the case of DER recursively embedded in BIT/OCTET STRINGs. So in this case you'd have to know you are expecting a data structure wrapped in a bitstring at a particular offset:
-strparse offset parse the contents octets of the ASN.1 object starting at offset. This option can be used multiple times to "drill down" into a nested structure.
with your example, we can use asn1parse to find the offset:
$ wget https://raw.githubusercontent.com/google/der-ascii/master/samples/cert.txt
--2019-04-24 15:15:27-- https://raw.githubusercontent.com/google/der-ascii/master/samples/cert.txt
Résolution de raw.githubusercontent.com (raw.githubusercontent.com)… 151.101.136.133
Connexion à raw.githubusercontent.com (raw.githubusercontent.com)|151.101.136.133|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 3079 (3,0K) [text/plain]
Sauvegarde en : « cert.txt »
cert.txt 100%[======================================================================================================>] 3,01K --.-KB/s ds 0s
2019-04-24 15:15:27 (6,05 MB/s) — « cert.txt » sauvegardé [3079/3079]
$ ascii2der -i cert.txt -o cert.der
$ openssl asn1parse -in cert.der -inform der -dump
0:d=0 hl=4 l= 600 cons: SEQUENCE
4:d=1 hl=4 l= 449 cons: SEQUENCE
8:d=2 hl=2 l= 3 cons: cont [ 0 ]
10:d=3 hl=2 l= 1 prim: INTEGER :02
13:d=2 hl=2 l= 9 prim: INTEGER :FBB04C2EAB109B0C
24:d=2 hl=2 l= 13 cons: SEQUENCE
26:d=3 hl=2 l= 9 prim: OBJECT :sha1WithRSAEncryption
37:d=3 hl=2 l= 0 prim: NULL
39:d=2 hl=2 l= 69 cons: SEQUENCE
41:d=3 hl=2 l= 11 cons: SET
43:d=4 hl=2 l= 9 cons: SEQUENCE
45:d=5 hl=2 l= 3 prim: OBJECT :countryName
50:d=5 hl=2 l= 2 prim: PRINTABLESTRING :AU
54:d=3 hl=2 l= 19 cons: SET
56:d=4 hl=2 l= 17 cons: SEQUENCE
58:d=5 hl=2 l= 3 prim: OBJECT :stateOrProvinceName
63:d=5 hl=2 l= 10 prim: UTF8STRING :Some-State
75:d=3 hl=2 l= 33 cons: SET
77:d=4 hl=2 l= 31 cons: SEQUENCE
79:d=5 hl=2 l= 3 prim: OBJECT :organizationName
84:d=5 hl=2 l= 24 prim: UTF8STRING :Internet Widgits Pty Ltd
110:d=2 hl=2 l= 30 cons: SEQUENCE
112:d=3 hl=2 l= 13 prim: UTCTIME :140423205040Z
127:d=3 hl=2 l= 13 prim: UTCTIME :170422205040Z
142:d=2 hl=2 l= 69 cons: SEQUENCE
144:d=3 hl=2 l= 11 cons: SET
146:d=4 hl=2 l= 9 cons: SEQUENCE
148:d=5 hl=2 l= 3 prim: OBJECT :countryName
153:d=5 hl=2 l= 2 prim: PRINTABLESTRING :AU
157:d=3 hl=2 l= 19 cons: SET
159:d=4 hl=2 l= 17 cons: SEQUENCE
161:d=5 hl=2 l= 3 prim: OBJECT :stateOrProvinceName
166:d=5 hl=2 l= 10 prim: UTF8STRING :Some-State
178:d=3 hl=2 l= 33 cons: SET
180:d=4 hl=2 l= 31 cons: SEQUENCE
182:d=5 hl=2 l= 3 prim: OBJECT :organizationName
187:d=5 hl=2 l= 24 prim: UTF8STRING :Internet Widgits Pty Ltd
213:d=2 hl=3 l= 159 cons: SEQUENCE
216:d=3 hl=2 l= 13 cons: SEQUENCE
218:d=4 hl=2 l= 9 prim: OBJECT :rsaEncryption
229:d=4 hl=2 l= 0 prim: NULL
231:d=3 hl=3 l= 141 prim: BIT STRING
0000 - 00 30 81 89 02 81 81 00-d8 2b c8 a6 32 e4 62 ff .0.......+..2.b.
0010 - 4d f3 d0 ad 59 8b 45 a7-bd f1 47 bf 09 58 7b 22 M...Y.E...G..X{"
0020 - bd 35 ae 97 25 86 94 a0-80 c0 b4 1f 76 91 67 46 .5..%.......v.gF
0030 - 31 d0 10 84 b7 22 1e 70-23 91 72 c8 e9 6d 79 3a 1....".p#.r..my:
0040 - 85 77 80 0f c4 95 16 75-c5 4a 71 4c c8 63 3f a3 .w.....u.JqL.c?.
0050 - f2 63 9c 2a 4f 9a fa cb-c1 71 6e 28 85 28 a0 27 .c.*O....qn(.(.'
0060 - 1e 65 1c ae 07 d5 5b 6f-2d 43 ed 2b 90 b1 8c af .e....[o-C.+....
0070 - 24 6d ae e9 17 3a 05 c1-bf b8 1c ae 65 3b 1b 58 $m...:......e;.X
0080 - c2 d9 ae d6 aa 67 88 f1-02 03 01 00 01 .....g.......
375:d=2 hl=2 l= 80 cons: cont [ 3 ]
377:d=3 hl=2 l= 78 cons: SEQUENCE
379:d=4 hl=2 l= 29 cons: SEQUENCE
381:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Subject Key Identifier
386:d=5 hl=2 l= 22 prim: OCTET STRING
0000 - 04 14 8b 75 d5 ac cb 08-be 0e 1f 65 b7 fa 56 be ...u.......e..V.
0010 - 6c a7 75 da 85 af l.u...
410:d=4 hl=2 l= 31 cons: SEQUENCE
412:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Authority Key Identifier
417:d=5 hl=2 l= 24 prim: OCTET STRING
0000 - 30 16 80 14 8b 75 d5 ac-cb 08 be 0e 1f 65 b7 fa 0....u.......e..
0010 - 56 be 6c a7 75 da 85 af- V.l.u...
443:d=4 hl=2 l= 12 cons: SEQUENCE
445:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Basic Constraints
450:d=5 hl=2 l= 5 prim: OCTET STRING
0000 - 30 03 01 01 ff 0....
457:d=1 hl=2 l= 13 cons: SEQUENCE
459:d=2 hl=2 l= 9 prim: OBJECT :sha1WithRSAEncryption
470:d=2 hl=2 l= 0 prim: NULL
472:d=1 hl=3 l= 129 prim: BIT STRING
0000 - 00 3b e8 78 6d 95 d6 3d-6a f7 13 19 2c 1b c2 88 .;.xm..=j...,...
0010 - ae 22 ab f4 8d 32 f5 7c-71 67 cf 2d d1 1c c2 c3 ."...2.|qg.-....
0020 - 87 e2 e9 be 89 5c e4 34-ab 48 91 c2 3f 95 ae 2b .....\.4.H..?..+
0030 - 47 9e 25 78 6b 4f 9a 10-a4 72 fd cf f7 02 0c b0 G.%xkO...r......
0040 - 0a 08 a4 5a e2 e5 74 7e-11 1d 39 60 6a c9 1f 69 ...Z..t~..9`j..i
0050 - f3 2e 63 26 dc 9e ef 6b-7a 0a e1 54 57 98 aa 72 ..c&...kz..TW..r
0060 - 91 78 04 7e 1f 8f 65 4d-1f 0b 12 ac 9c 24 0f 84 .x.~..eM.....$..
0070 - 14 1a 55 2d 1f bb f0 9d-09 b2 08 5c 59 32 65 80 ..U-.......\Y2e.
0080 - 26 &
There's two bitstrings in there, at byte 231 and 472; we can unpack them with:
$ openssl asn1parse -in cert.der -inform der -dump -strparse 231
0:d=0 hl=3 l= 137 cons: SEQUENCE
3:d=1 hl=3 l= 129 prim: INTEGER :D82BC8A632E462FF4DF3D0AD598B45A7BDF147BF09587B22BD35AE97258694A080C0B41F7691674631D01084B7221E70239172C8E96D793A8577800FC4951675C54A714CC8633FA3F2639C2A4F9AFACBC1716E288528A0271E651CAE07D55B6F2D43ED2B90B18CAF246DAEE9173A05C1BFB81CAE653B1B58C2D9AED6AA6788F1
135:d=1 hl=2 l= 3 prim: INTEGER :010001
Parsing the second one errors, because it is a literal bitstring and asn1parse doesn't support outputting primitives directly (I don't understand why OpenSSL can't support this, but they refuse it):
$ openssl asn1parse -in cert.der -inform der -dump -strparse 472
Error in encoding
4493112940:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:asn1_lib.c:157:
It turns out asn1parse has its own ASCIIified format. Encoding is openssl asn1parse -genconf or openssl asn1parse -strconf1, but the input format isn't the same as the output format (so, thanks for der2ascii!!). The input format is documented in ASN1_generate_nconf(3), and it has special WRAP types for these cases:
OCTWRAP, SEQWRAP, SETWRAP, BITWRAP The following structure is surrounded by an OCTET STRING, a SEQUENCE, a SET or a BIT STRING respectively. For a BIT STRING the number of unused bits is set to zero.
We can try this out with the example from the manpage:
$ cat pubkey.asn1.cnf
# Start with a SEQUENCE
asn1=SEQUENCE:pubkeyinfo
# pubkeyinfo contains an algorithm identifier and the public key wrapped
# in a BIT STRING
[pubkeyinfo]
algorithm=SEQUENCE:rsa_alg
pubkey=BITWRAP,SEQUENCE:rsapubkey
# algorithm ID for RSA is just an OID and a NULL
[rsa_alg]
parameter=NULL
algorithm=OID:rsaEncryption
# Actual public key: modulus and exponent
[rsapubkey]
n=INTEGER:0xBB6FE79432CC6EA2D8F970675A5A87BFBE1AFF0BE63E879F2AFFB93644D4D2C6D000430DEC66ABF47829E74B8C5108623A1C0EE8BE217B3AD8D36D5EB4FCA1D9
e=INTEGER:0x010001
$
$ openssl asn1parse -genconf pubkey.asn1.cnf -noout -out pubkey.asn1.der
$ openssl asn1parse -in pubkey.asn1.der -inform der
0:d=0 hl=2 l= 92 cons: SEQUENCE
2:d=1 hl=2 l= 13 cons: SEQUENCE
4:d=2 hl=2 l= 0 prim: NULL
6:d=2 hl=2 l= 9 prim: OBJECT :rsaEncryption
17:d=1 hl=2 l= 75 prim: BIT STRING
$ openssl asn1parse -in pubkey.asn1.der -inform der -strparse 17
0:d=0 hl=2 l= 72 cons: SEQUENCE
2:d=1 hl=2 l= 65 prim: INTEGER :BB6FE79432CC6EA2D8F970675A5A87BFBE1AFF0BE63E879F2AFFB93644D4D2C6D000430DEC66ABF47829E74B8C5108623A1C0EE8BE217B3AD8D36D5EB4FCA1D9
69:d=1 hl=2 l= 3 prim: INTEGER :010001
Weirdly, and probably the source of years of confusion, -strparse will accept either a bitwrapped address of the; it doesn't see a string type, it acts as if it was passed -offset rather than -strparse. And that means that manually skipping past the header will output identical results:
$ openssl asn1parse -in pubkey.asn1.der -inform der -strparse 17
0:d=0 hl=2 l= 72 cons: SEQUENCE
2:d=1 hl=2 l= 65 prim: INTEGER :BB6FE79432CC6EA2D8F970675A5A87BFBE1AFF0BE63E879F2AFFB93644D4D2C6D000430DEC66ABF47829E74B8C5108623A1C0EE8BE217B3AD8D36D5EB4FCA1D9
69:d=1 hl=2 l= 3 prim: INTEGER :010001
$ openssl asn1parse -in pubkey.asn1.der -inform der -strparse 20
0:d=0 hl=2 l= 72 cons: SEQUENCE
2:d=1 hl=2 l= 65 prim: INTEGER :BB6FE79432CC6EA2D8F970675A5A87BFBE1AFF0BE63E879F2AFFB93644D4D2C6D000430DEC66ABF47829E74B8C5108623A1C0EE8BE217B3AD8D36D5EB4FCA1D9
69:d=1 hl=2 l= 3 prim: INTEGER :010001
I wonder if BITWRAP is denoted somehow differently than BITSTRING. I changed that file:
$ diff -u pubkey*.asn1.cnf
--- pubkey-nowrap.asn1.cnf 2019-04-24 15:59:53.000000000 -0400
+++ pubkey.asn1.cnf 2019-04-24 16:00:07.000000000 -0400
@@ -5,7 +5,7 @@
# in a BIT STRING
[pubkeyinfo]
algorithm=SEQUENCE:rsa_alg
-pubkey=SEQUENCE:rsapubkey
+pubkey=BITWRAP,SEQUENCE:rsapubkey
# algorithm ID for RSA is just an OID and a NULL
[rsa_alg]
and regenerated:
$ openssl asn1parse -genconf pubkey-nowrap.asn1.cnf -noout -out pubkey-nowrap.asn1.der
$ openssl asn1parse -in pubkey-nowrap.asn1.der -inform der -dump
0:d=0 hl=2 l= 89 cons: SEQUENCE
2:d=1 hl=2 l= 13 cons: SEQUENCE
4:d=2 hl=2 l= 0 prim: NULL
6:d=2 hl=2 l= 9 prim: OBJECT :rsaEncryption
17:d=1 hl=2 l= 72 cons: SEQUENCE
19:d=2 hl=2 l= 65 prim: INTEGER :BB6FE79432CC6EA2D8F970675A5A87BFBE1AFF0BE63E879F2AFFB93644D4D2C6D000430DEC66ABF47829E74B8C5108623A1C0EE8BE217B3AD8D36D5EB4FCA1D9
86:d=2 hl=2 l= 3 prim: INTEGER :010001
$ openssl asn1parse -in pubkey-nowrap.asn1.der -inform der -strparse 17
0:d=0 hl=2 l= 72 cons: SEQUENCE
2:d=1 hl=2 l= 65 prim: INTEGER :BB6FE79432CC6EA2D8F970675A5A87BFBE1AFF0BE63E879F2AFFB93644D4D2C6D000430DEC66ABF47829E74B8C5108623A1C0EE8BE217B3AD8D36D5EB4FCA1D9
69:d=1 hl=2 l= 3 prim: INTEGER :010001
The difference is three bytes, 03 4b 00, inserted at offset 0x11 == 17:
$ ls -l pubkey*.asn1.der
-rw-r--r-- 1 kousu staff 91 24 avr 15:58 pubkey-nowrap.asn1.der
-rw-r--r-- 1 kousu staff 94 24 avr 15:54 pubkey.asn1.der
$ hexdump -C pubkey.asn1.der
00000000 30 5c 30 0d 05 00 06 09 2a 86 48 86 f7 0d 01 01 |0\0.....*.H.....|
00000010 01 03 4b 00 30 48 02 41 00 bb 6f e7 94 32 cc 6e |..K.0H.A..o..2.n|
00000020 a2 d8 f9 70 67 5a 5a 87 bf be 1a ff 0b e6 3e 87 |...pgZZ.......>.|
00000030 9f 2a ff b9 36 44 d4 d2 c6 d0 00 43 0d ec 66 ab |.*..6D.....C..f.|
00000040 f4 78 29 e7 4b 8c 51 08 62 3a 1c 0e e8 be 21 7b |.x).K.Q.b:....!{|
00000050 3a d8 d3 6d 5e b4 fc a1 d9 02 03 01 00 01 |:..m^.........|
0000005e
$ hexdump -C pubkey-nowrap.asn1.der
00000000 30 59 30 0d 05 00 06 09 2a 86 48 86 f7 0d 01 01 |0Y0.....*.H.....|
00000010 01 30 48 02 41 00 bb 6f e7 94 32 cc 6e a2 d8 f9 |.0H.A..o..2.n...|
00000020 70 67 5a 5a 87 bf be 1a ff 0b e6 3e 87 9f 2a ff |pgZZ.......>..*.|
00000030 b9 36 44 d4 d2 c6 d0 00 43 0d ec 66 ab f4 78 29 |.6D.....C..f..x)|
00000040 e7 4b 8c 51 08 62 3a 1c 0e e8 be 21 7b 3a d8 d3 |.K.Q.b:....!{:..|
00000050 6d 5e b4 fc a1 d9 02 03 01 00 01 |m^.........|
0000005b
\x03 is, presumably, the BIT STRING type, 0x4b = 75 is the length, and \x00 is the weird leading null that is in all these examples. I would say the leading null is the clue that there's an embedded struct, but the second cert.txt bitstring also has a leading null (but that null gets skipped when parsed back by openssl x509 -in cert.pem -text). Do all BIT STRINGS need a leading null?
There doesn't really seem to be anything in the serialized form that signals BITWRAP should investigated, so you're stuck with your current algorithm of guessing-and-backtracking.
I'd suggest der2ascii should not be guessing by default because it is incompatible with OpenSSL.
The best long term solution is for your format to grow a BITWRAP type and to have a flag analogous to -strparse to cause it to try to recurse downwards; maybe unlike openssl's offset jump + parse attempt, your flag could be the turn on the heuristic flag.
Here's a simpler test case that demonstrates the problem. Decoding is not 1:1:
$ cat icantbelieveitsnotyubikey.pub.txt
SEQUENCE {
SEQUENCE {
# ecPublicKey
OBJECT_IDENTIFIER { 1.2.840.10045.2.1 }
# secp256r1
OBJECT_IDENTIFIER { 1.2.840.10045.3.1.7 }
}
BIT_STRING {
`00`
`043fd38bc5df75d6cae055cf8160b23277d295f59a8e34a2b0dc69b81af90a1c042c02c95580e69e8837e752eca1395e30a9202283ec1a85f8daa78ade833c752b`
}
}
$ diff -u icantbelieveitsnotyubikey.pub.txt <(ascii2der < icantbelieveitsnotyubikey.pub.txt | der2ascii)
--- icantbelieveitsnotyubikey.pub.txt 2019-04-24 16:46:12.000000000 -0400
+++ /dev/fd/63 2019-04-24 16:49:10.000000000 -0400
@@ -7,7 +7,6 @@
}
BIT_STRING {
`00`
- `043fd38bc5df75d6cae055cf8160b23277d295f59a8e34a2b0dc69b81af90a1c042c02c95580e69e8837e752eca1395e30a9202283ec1a85f8daa78ade833c752b`
+ OCTET_STRING { `d38bc5df75d6cae055cf8160b23277d295f59a8e34a2b0dc69b81af90a1c042c02c95580e69e8837e752eca1395e30a9202283ec1a85f8daa78ade833c752b` }
}
}
Although encoding is alright:
$ diff -u icantbelieveitsnotyubikey.pub.der <(der2ascii < icantbelieveitsnotyubikey.pub.der | ascii2der)
$
I would say the leading null is the clue that there's an embedded struct, but the second cert.txt bitstring also has a leading null (but that null gets skipped when parsed back by openssl x509 -in cert.pem -text). Do all BIT STRINGS need a leading null?
No, the leading null is not a clue that there's an embedded struct. It simply means that there are a multiple of 8 bits.
In general, BIT STRINGs need not be a whole number of bytes (aka octets) long. That's the difference between BIT STRING and OCTET STRING. The way you encode BIT STRINGs in DER is to first pad it up to a whole number of bytes, then prepend a byte containing the number of padding bits you added. So, for instance, a 13-bit BIT STRING will be 3 bytes long. First a byte containing the value 3, then two bytes containing 13+3 = 16 bits.
It sounds like you're making a number of guesses about ASN.1 from OpenSSL's ad-hoc config file language. I would recommend familiarizing yourself with how ASN.1 and DER work. The actual specs are available online, but they're a little opaque. This is a good guide.
http://luca.ntop.org/Teaching/Appunti/asn1.html
The best long term solution is for your format to grow a BITWRAP type and to have a flag analogous to -strparse to cause it to try to recurse downwards; maybe unlike openssl's offset jump + parse attempt, your flag could be the turn on the heuristic flag.
No, there is no need for a BITWRAP type or anything of the sort. BITWRAP, in this tool, is simply:
BIT_STRING { `00` insert your contents here }
Nor would it help because, again, the problem is this aspect of ASN.1 is not self-describing. BITWRAP only makes sense in OpenSSL due to how the config file language works. Again, I would recommend familiarizing yourself with how ASN.1 works.
I'd suggest der2ascii should not be guessing by default because it is incompatible with OpenSSL.
No, that would make der2ascii useless for printing certificates, where there are many ASN.1 structures embedded in OCTET STRINGs. Matching openssl asn1parse is not a goal of this project. If openssl asn1parse is more useful for you, you should just use that. For this project's goals, under-recursing is fatal, while over-recursing is annoying but not a disaster because the opaque byte string was likely not editable anyway.
One thing that might work is to add a comment in front of guesses containing the byte string version, so that if it guesses wrong, you can uncomment that and continue on editing things.
Although encoding is alright:
Right. As I said, it's a valid decode. It's not the most useful one, but there is no correctness failure. As documented, der2ascii is fundamentally a guess.
I don't want to use asn1parse. It's arcane and difficult. I like der2ascii much much more. But I guess I will have to figure out something else for my use case because I'm trying to use arbitrary ECC keys that I don't control and I can't have them misread like that.
Is your use case just printing things out, or are you trying to programmatically run on the output of der2ascii? I'm willing to toy with making the human-readable output of der2ascii more useful when it guesses wrong, but the tool does explicitly disavow keeping the der2ascii output stable, only the other direction. (Otherwise I would have no ability to tweak it with, e.g., new OIDs or syntax.)
https://github.com/google/der-ascii/#backwards-compatibility
I'm trying to load, store, and pass keys on a system with minimal dependencies, so I'm using bash and was checking my work with der2ascii and thinking of using it inline because it should be much more reliable than asn1parse. I suppose I should just give up on bash and write in Go at this point.
For anyone coming this way later, I discovered that the confusingly written openssl asn1parse -i /dev/stdin -inform DER -offset $N -noout -out /dev/stdout | xxd -p will extract the data at offset $N fully, and in the case of BITWRAPPED structs you can use -strparse $N instead and it'll automatically eat away the string header. -noout and -out are not the same; one silences asn1parse's asciification, the other causes raw binary to come out.
Thank you for your time.
I see. Yeah, I wouldn't recommend doing cryptography in bash. der2ascii also won't check things like whether the public key is actually on the curve.