crash importing keyring

[philpennock]

wotmate's make-sqlitedb.py tool created a DB just fine from the limited set of keys on my laptop, but when I copied pubring.kbx from the normal signing box to here, it crashed.

There's nothing in the trace to identify the key which caused issues (a catch/raise X from ... might help with that?) but with it crashing after 3hrs50m it's not something I'm going to repeat immediately.

I suspect that the issue is simply a Latin1 encoding of a name: 0xf6 is ö (in both ASCII and as Unicode codepoint).

[ ... ]
Loaded 2290 pubkeys
Loading uid and signature data
Running /usr/local/bin/gpg --with-colons --list-sigs...
gpg: WARNING: unsafe permissions on homedir '/Users/pdp/src/security/openpgp/wotmate/pdp/bb'
gpg: Note: signatures using the MD5 algorithm are rejected
gpg: public key E204BE5C475470DA is 10 days newer than the signature
gpg: public key E204BE5C475470DA is 10 days newer than the signature
gpg: public key E204BE5C475470DA is 10 days newer than the signature
gpg: public key E204BE5C475470DA is 10 days newer than the signature
Traceback (most recent call last):
  File "../../make-sqlitedb.py", line 311, in <module>
    keyring_populate_uid_sig_data(cursor, kr_map)
  File "../../make-sqlitedb.py", line 127, in keyring_populate_uid_sig_data
    fields = wotmate.gpg_get_fields(line)
  File "/Users/pdp/src/security/openpgp/wotmate/wotmate/__init__.py", line 111, in gpg_get_fields
    line = bline.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 48: invalid start byte
../../make-sqlitedb.py --gpgbin==gpg --gnupghome=.  734.16s user 2411.34s system 22% cpu 3:50:32.27 total

[mricon]

Wow, 3.5 hours. 😞

I expect it's all the shelling out to call gpg -- I gotta see if we can query pubring.kbx more efficiently than that. I'll commit a change that should do a better job dealing with malformed unicode, standby.

[mricon]

Well, this doesn't fix the slowness (it's all in waiting for gpg --list-sigs --with-colons to complete, so not much that I can do there, unless I find a way to speed that up), but this should not crash on finding malformed utf-8.

[philpennock]

I've not checked to see if it affects data that you need here, but I've had good luck with --fast-list-mode in the past.

gpg --with-colons --fast-list-mode --batch --list-keys </dev/null

[philpennock]

Well, you want --list-sigs, not the example I happened to grab from a script lying around, duh. But the point stands:

% time GNUPGHOME=. gpg --with-colons --fast-list-mode --batch --list-sigs </dev/null  | wc -l
  966434
GNUPGHOME=. gpg --with-colons --fast-list-mode --batch --list-sigs < /dev/nul  28.41s user 0.48s system 98% cpu 29.429 total

[mricon]

Thanks for the suggestion -- I rewrote keyring import using --fast-list-mode and it should now take seconds even on large keyrings.