Error with database version 118
Opened this issue · 11 comments
Hello and thank you for your job 🙂
I already ran signal2html with success but this time it's not working 😕
I've got this :
$ signal2html -i signal-backup-2021-11-15-14-02-50 -o signal-backup-2021-11-15-14-02-50_html
Found untested Signal database version: 118.
Group for recipient 1267 is '__textsecure_group__!2bdafcb85cd87e830306786cd49e0e9d' which does not exist.
Group for recipient 1268 is '__textsecure_group__!686d5b59b8720222ccb7864d99145ec3' which does not exist.
Group for recipient 1269 is '__textsecure_group__!8b6b00a62bfef9ff5e300a3eb348533a' which does not exist.
Group for recipient 1270 is '__textsecure_group__!bb2abe181e3f8e72efbbe96743c9585f' which does not exist.
Group for recipient 1271 is '__textsecure_group__!bc7acbecf4c8130a125a311bef4e292c' which does not exist.
Group for recipient 1294 is '__textsecure_group__!b7f2a230bd2f997b918066b89ae22ca0' which does not exist.
Traceback (most recent call last):
File "/home/martin/.local/bin/signal2html", line 8, in <module>
sys.exit(main())
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/__main__.py", line 15, in main
sys.exit(realmain())
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/ui.py", line 35, in main
process_backup(args.input_dir, args.output_dir)
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/core.py", line 257, in process_backup
query = db.execute("SELECT _id, recipient_ids FROM thread")
sqlite3.OperationalError: no such column: recipient_ids
Unfortunately I can't help more, I don't understand anythings about Python 😓 🐍
But I am ok to contribute with beers 🍻 (or something else)
Thanks for your help 🙂
By reading database (with DB Browser for SQLite) I see the structure of the table "thread" :
CREATE TABLE "thread" (_id INTEGER PRIMARY KEY AUTOINCREMENT, date INTEGER DEFAULT 0, thread_recipient_id INTEGER, message_count INTEGER DEFAULT 0, snippet TEXT, snippet_charset INTEGER DEFAULT 0, snippet_type INTEGER DEFAULT 0, snippet_uri TEXT DEFAULT NULL, snippet_content_type INTEGER DEFAULT NULL, snippet_extras TEXT DEFAULT NULL, read INTEGER DEFAULT 1, type INTEGER DEFAULT 0, error INTEGER DEFAULT 0, archived INTEGER DEFAULT 0, status INTEGER DEFAULT 0, expires_in INTEGER DEFAULT 0, last_seen INTEGER DEFAULT 0, has_sent INTEGER DEFAULT 0, delivery_receipt_count INTEGER DEFAULT 0, read_receipt_count INTEGER DEFAULT 0, unread_count INTEGER DEFAULT 0, last_scrolled INTEGER DEFAULT 0, pinned INTEGER DEFAULT 0)
I can't see recipient_ids but thread_recipient_id. Is that the problem ?
Can I help in another way ?
Thanks
Hi @Martin-bcd,
What version of signal2html are you using? From your error message above it looks like you may be using an older version. Try updating it with pip install -U signal2html, and let me know if that solves the problem! :)
Thanks, it seems to work 😄
I am on Manjaro and I am used to have all my packages updated in one time with Pacman but I forgot that I installed it with pip 😅
👍
OK, As soon as I closed the issue, it failed again ! 😓
It worked longer but it failed :
$ signal2html -V
0.2.6
$ signal2html -i signal-backup-2021-11-15-14-02-50 -o signal-backup-2021-11-15-14-02-50_html
Found untested Signal database version: 118.
Group for recipient 1267 is '__textsecure_group__!2bdafcb85cd87e830306786cd49e0e9d' which does not exist.
Group for recipient 1268 is '__textsecure_group__!686d5b59b8720222ccb7864d99145ec3' which does not exist.
Group for recipient 1269 is '__textsecure_group__!8b6b00a62bfef9ff5e300a3eb348533a' which does not exist.
Group for recipient 1270 is '__textsecure_group__!bb2abe181e3f8e72efbbe96743c9585f' which does not exist.
Group for recipient 1271 is '__textsecure_group__!bc7acbecf4c8130a125a311bef4e292c' which does not exist.
Group for recipient 1294 is '__textsecure_group__!b7f2a230bd2f997b918066b89ae22ca0' which does not exist.
Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_1834_1586808825833.bin'. Maybe it was deleted or never downloaded.
Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_690_1531739530460.bin'. Maybe it was deleted or never downloaded.
Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_691_1531739538200.bin'. Maybe it was deleted or never downloaded.
Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_692_1531739548574.bin'. Maybe it was deleted or never downloaded.
Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_2454_1604072295244.bin'. Maybe it was deleted or never downloaded.
Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_2455_1604072304388.bin'. Maybe it was deleted or never downloaded.
Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_1934_1588425451506.bin'. Maybe it was deleted or never downloaded.
Failed to load group update data (v2) for message 21239: __init__() missing 1 required positional argument: 'change'
Traceback (most recent call last):
File "/home/martin/.local/bin/signal2html", line 8, in <module>
sys.exit(main())
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/__main__.py", line 15, in main
sys.exit(realmain())
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/ui.py", line 35, in main
process_backup(args.input_dir, args.output_dir)
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/core.py", line 672, in process_backup
dump_thread(t, output_dir)
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/html.py", line 243, in dump_thread
event_data = format_event_data_group_update(
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/html.py", line 125, in format_event_data_group_update
if data.change_by:
AttributeError: 'list' object has no attribute 'change_by'
Always the same warning on top : Found untested Signal database version: 118.
( I was in version 0.2.1 before and now 0.2.6)
It is a different issue but always with database version 118 so I keep the same ticket !
Thanks for testing this @Martin-bcd! 👍 Glad to hear we got one issue fixed at least!
It sounds like the issue you have now is the same as in #52. If you like, you can try the fix that I've suggested there to see if that solves it (that would help me to figure out the issue).
The warning about the untested Signal database version means that you're on a version that I've not tried yet, so I can't guarantee that it works (which, in this case, it doesn't). As soon as we figure out how to fix this and we get it to work, I'll update the code and remove that warning :)
OK. Thank.
On my previous post I said it failed and it's correct but I got lots of data and maybe it was very close to the end
This is what I am doing now :
$ pip uninstall signal2html
(...)
$ pip install -U git+https://github.com/GjjvdBurg/signal2html.git@bugfix/group_update_v2
(...)
$ signal2html -V
0.2.6
signal2html -i signal-backup-2021-11-15-14-02-50 -o signal-backup-2021-11-15-14-02-50_html2
2021-11-22 23:06:55 | INFO - This is signal2html version 0.2.6
2021-11-22 23:06:55 | INFO - Using database version 118.
2021-11-22 23:06:55 | WARNING - Found untested Signal database version: 118.
2021-11-22 23:06:55 | INFO - Group for recipient 1259 is '__signal_mms_group__!035c1f6593090a168a11b5f8e0c06660' and will be called by group id using name 'Group 2'.
2021-11-22 23:06:55 | INFO - Group for recipient 1260 is '__signal_mms_group__!0879969925e89a47117a5a5e9e093647' and will be called by group id using name 'Group 1'.
2021-11-22 23:06:55 | INFO - Group for recipient 1261 is '__signal_mms_group__!2bdafcb85cd87e830306786cd49e0e9d' and will be called by group id using name 'Group 5'.
2021-11-22 23:06:55 | INFO - Group for recipient 1262 is '__signal_mms_group__!455d1b2014869e4f75aa6cfbe2d58a70' and will be called by group id using name 'Group 4'.
2021-11-22 23:06:55 | INFO - Group for recipient 1263 is '__signal_mms_group__!533e2936c7de17a62dc0c6d09621e476' and will be called by group id using name 'Group 3'.
2021-11-22 23:06:55 | INFO - Group for recipient 1264 is '__signal_mms_group__!8b6b00a62bfef9ff5e300a3eb348533a' and will be called by group id using name 'Group 9'.
2021-11-22 23:06:55 | INFO - Group for recipient 1265 is '__signal_mms_group__!a813d75da32dea4933f5ed2fa25139f0' and will be called by group id using name 'Group 11'.
2021-11-22 23:06:55 | INFO - Group for recipient 1266 is '__signal_mms_group__!bc7acbecf4c8130a125a311bef4e292c' and will be called by group id using name 'Group 10'.
2021-11-22 23:06:55 | WARNING - Group for recipient 1267 is '__textsecure_group__!2bdafcb85cd87e830306786cd49e0e9d' which does not exist.
2021-11-22 23:06:55 | WARNING - Group for recipient 1268 is '__textsecure_group__!686d5b59b8720222ccb7864d99145ec3' which does not exist.
2021-11-22 23:06:55 | WARNING - Group for recipient 1269 is '__textsecure_group__!8b6b00a62bfef9ff5e300a3eb348533a' which does not exist.
2021-11-22 23:06:55 | WARNING - Group for recipient 1270 is '__textsecure_group__!bb2abe181e3f8e72efbbe96743c9585f' which does not exist.
2021-11-22 23:06:55 | WARNING - Group for recipient 1271 is '__textsecure_group__!bc7acbecf4c8130a125a311bef4e292c' which does not exist.
2021-11-22 23:06:55 | INFO - Group for recipient 1282 is '__signal_mms_group__!5efd34cd8812e3917cd485cfa52844dd' and will be called by group id using name 'Group 14'.
2021-11-22 23:06:55 | INFO - Group for recipient 1283 is '__signal_mms_group__!686d5b59b8720222ccb7864d99145ec3' and will be called by group id using name 'Group 8'.
2021-11-22 23:06:55 | INFO - Group for recipient 1284 is '__signal_mms_group__!8e9f2c8e26d8a1b3d83d1f30cddb6205' and will be called by group id using name 'Group 16'.
2021-11-22 23:06:55 | INFO - Group for recipient 1285 is '__signal_mms_group__!967b00f2a55eec4cbf941d862a50f6e6' and will be called by group id using name 'Group 7'.
2021-11-22 23:06:55 | INFO - Group for recipient 1286 is '__signal_mms_group__!ad26c35cb0b9e3eb11dcb3c911fabcff' and will be called by group id using name 'Group 12'.
2021-11-22 23:06:55 | INFO - Group for recipient 1287 is '__signal_mms_group__!b7f2a230bd2f997b918066b89ae22ca0' and will be called by group id using name 'Group 17'.
2021-11-22 23:06:55 | INFO - Group for recipient 1288 is '__signal_mms_group__!bb2abe181e3f8e72efbbe96743c9585f' and will be called by group id using name 'Group 6'.
2021-11-22 23:06:55 | INFO - Group for recipient 1289 is '__signal_mms_group__!e2f88b3443872b5969992b1818768677' and will be called by group id using name 'Group 13'.
2021-11-22 23:06:55 | INFO - Group for recipient 1290 is '__signal_mms_group__!e5f5c4f56f8edfcd6db5ac8e72b45b3d' and will be called by group id using name 'Group 15'.
2021-11-22 23:06:55 | WARNING - Group for recipient 1294 is '__textsecure_group__!b7f2a230bd2f997b918066b89ae22ca0' which does not exist.
2021-11-22 23:09:38 | WARNING - Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_1834_1586808825833.bin'. Maybe it was deleted or never downloaded?
2021-11-22 23:09:50 | WARNING - Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_690_1531739530460.bin'. Maybe it was deleted or never downloaded?
2021-11-22 23:09:50 | WARNING - Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_691_1531739538200.bin'. Maybe it was deleted or never downloaded?
2021-11-22 23:09:50 | WARNING - Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_692_1531739548574.bin'. Maybe it was deleted or never downloaded?
2021-11-22 23:09:53 | WARNING - Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_2454_1604072295244.bin'. Maybe it was deleted or never downloaded?
2021-11-22 23:09:53 | WARNING - Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_2455_1604072304388.bin'. Maybe it was deleted or never downloaded?
2021-11-22 23:10:24 | WARNING - Couldn't find attachment '/run/media/martin/72a42117-f893-4709-9c3f-f6d1001d9ab4/XZ/signal-backup-2021-11-15-14-02-50/Attachment_1934_1588425451506.bin'. Maybe it was deleted or never downloaded?
2021-11-22 23:10:40 | WARNING - Failed to load group update data (v2) for message 21239:
Message body: 'b'\n$\n \x83\xb2\xfd\xf2\xf0\'?\x8a2%z\xd4\xe7\xdb\x1cy\x87\xa7\x00\x14au\xb0k\xa2\x8c[\xac\xde\x9dl\xda\x10\x00\x1a\xb3\x07\x12\x1bAnniv surprise de Reno \xf0\x9f\x98\x9c"\x00*\x04\x08\x02\x10\x02:6\n\x10\x0cP\x13K\xceiB\xb3\x90=\xd0\xe9\xab\xf9Z8\x10\x02\x1a \x83\xc2\x93\xb9\xe4\x96x\xf8\x93\x06\xf7tb\xd7\xf4\n\xadP\xfeA\x91\x1c\xa20\xffk_\\\x8a\xc5\xed\x8a:6\n\x10\xca\xe6J\x03\xcf\x8cMx\x80\x18NV\xaae\x84\x90\x10\x01\x1a }\x7fVFV\xf9O\xd1\xb6l\xb3\xaa\xba\x81\x14\xf4\xe1\x94\xfc\xe6\xb3Hn\x1e\xb1T\x8agT\xb0\x9d\x0fBp\n\x10\x92\xe4\x8c\xf2\xb5\\B\xa7\xae\x00\x97\xe3\x06<\x18c\x10\x01\x1a\x10\x0cP\x13K\xceiB\xb3\x90=\xd0\xe9\xab\xf9Z8 \xc9\xd2\x85\xec\xf5.*A\x00\x14\xe3]I:{hG\xab\x80L\x95yxe\xe3QHA\tk\r\xee\x9bQ\x88\x8d\x95.\xd6\xcdX\x18\xad-rr\xa4\xd2\xed&>x( \x1d\xb6C\xfd\x83\x0bA\x88gL\xa0\x8a\xff\x93\xc8\x9b\xef\xe0\x07Bp\n\x10p\xb4\xd1\x18S\x0fOt\x8f\xc0\xc9P\nVt\xd3\x10\x01\x1a\x10\x0cP\x13K\xceiB\xb3\x90=\xd0\xe9\xab\xf9Z8 \xc9\xd2\x85\xec\xf5.*A\x00z\xa2@k\xcf\x9b\xbc\x1e\xd5\xc0M\xef/B\xdf\x9c\x89_\x8a+Jr~_\x08|B7\x84\x82]%,\x1c\x81\x1a"{\xf1\xf2\xfe\x0b9#\xc0\xda\xb0y\xf9y\xd1\xfd bh\xef\x0b\x04\xa4@\x80uw3Bp\n\x10Z>\x7f\xa0\xce\xfdEo\xaa\xaf)|\'h\xda$\x10\x01\x1a\x10\x0cP\x13K\xceiB\xb3\x90=\xd0\xe9\xab\xf9Z8 \xc9\xd2\x85\xec\xf5.*A\x00 \xb9\xef\x03\x9a\x81O\xe0Y\xe1f\xf2\xb1\x94D\x16u\r\xd3\x17\xcbub\xecQ\x91\xafM\x07\xcc\x9d]\xc4\xd0\x1f\xec\xe0\xc7\xaa\x8a\rR \xd8\xa3H\x8c.\xb4\xc2Ev\xee\x0bM\xfa\x87!\xd34\x9a\xa1\x8b\x1eBp\n\x10\xee\x94.t\xa4\xb4M$\x93\x1f^\xdc\x93 U[\x10\x01\x1a\x10\x0cP\x13K\xceiB\xb3\x90=\xd0\xe9\xab\xf9Z8 \xc9\xd2\x85\xec\xf5.*A\x00\xdc\xcf\'\xba\xc3\xf7U\x1c_\x89A\x1f\xd6I\x9c\x81\xd8\x11X\xada\xe5\xfaU\x0f\x82\x8f\x15\xb3\xe5@Y\xc6\xdbE~r\xc9G\xa3_\xe4)\xd9\x04\x1c\xfc\xd2bY\xc0I\xa7\xe1-9\xb3L\x13\xf6\xb8\xb5\n\x17Bp\n\x10?Ub\xe8\xe6\x1fD\x12\x9b\x86;\xc2\x06=\x1ab\x10\x01\x1a\x10\x0cP\x13K\xceiB\xb3\x90=\xd0\xe9\xab\xf9Z8 \xc9\xd2\x85\xec\xf5.*A\x00\x0e\xe8\r\xc3*\x9d\xce6\x07\xf7m\x01\xd8\xb7\xa3\x0b\xa1\xf0\r,\x12Jk-\'\x84\x11\xbf\xdeH\x93+Rx\x8f\x9e\xa2f\\\n\x12O\x02\x10\x10o/\x88H\xfa{\xbd]\x1e\x9a&\\{ \xffq\x0e\xc57Bp\n\x10q\x86.s\x86\xdcL=\xad\\\x8c/\xd4\x9fq4\x10\x01\x1a\x10\x0cP\x13K\xceiB\xb3\x90=\xd0\xe9\xab\xf9Z8 \xc9\xd2\x85\xec\xf5.*A\x00\xc6*\x19\xcb\xae\xcd\x8e\xeb\x92\xc7\xbc\xf9\xff\x1c\x0b \x16?\xfa`4\xda\xcfb8\x15\xd3\x8aQo\xce+\x16\t\xa8\xdc\x90\xc3}U\t K\x1d2\x16\xad<\xe05\xdd\x1c,-r`\xcbt\x14\n\x8b\xfbmWBp\n\x10=\xb2\xb4\xd9\xf2\x1cI\xe8\x92j\x11\xc2\xf0\xdf\xb4%\x10\x01\x1a\x10\x0cP\x13K\xceiB\xb3\x90=\xd0\xe9\xab\xf9Z8 \xc9\xd2\x85\xec\xf5.*A\x00\x16\x1c\xb4-\x19$\xe44i4\xc7\xb1uyh\xc0\xfd\xf4$\xddHWT\'\xda\xad \xc3G8W\x12\xd6\xf0P5\xc8E\x001\x84\x1c0K\x17c\x0b\xf5\xd1\xf0&\xdc\x95\xca\x993\xf6*c\xa9|\xb8r<''
Error message: __init__() missing 1 required positional argument: 'change'
Traceback (most recent call last):
File "/home/martin/.local/bin/signal2html", line 8, in <module>
sys.exit(main())
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/__main__.py", line 20, in main
sys.exit(main())
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/ui.py", line 35, in main
process_backup(args.input_dir, args.output_dir)
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/core.py", line 685, in process_backup
dump_thread(t, output_dir)
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/html.py", line 243, in dump_thread
event_data = format_event_data_group_update(
File "/home/martin/.local/lib/python3.9/site-packages/signal2html/html.py", line 125, in format_event_data_group_update
if data.change_by:
AttributeError: 'NoneType' object has no attribute 'change_by'
Thank you :-)
the first run I got a folder with : 4 624 elements, 7,3 Go
the last one : 4 624 elements, 7,3 Go
So same result ??
Thanks for the additional testing @Martin-bcd! This helps to figure out what's going on.
@ericthegrey do you have any thoughts on this? It looks like the StructuredGroupDataV2 protobuf message has changed but I'm not sure where in the Signal source this is defined. The other issue is that None data isn't handled properly, but that's more a workaround I think. Thanks!
Hi, I can't look at this in details for some time, but it looks like the top level structure of this message is different (it might encapsulate the previous one). Perhaps the data type field has been updated to indicate this, or perhaps the change is implicit in the new database version (I haven't found a content upgrade procedure, however, but I looked only briefly).
@GjjvdBurg FYI - you can find the original proto for the previous structure here: https://github.com/signalapp/Signal-Android/blob/master/libsignal/service/src/main/proto/DecryptedGroups.proto#L69 and the one I suspect is now used at: https://github.com/signalapp/Signal-Android/blob/master/app/src/main/proto/Database.proto#L47
@GjjvdBurg - I would suggest writing some warning when outputting message data in logs, since people might not realize this could contain confidential message information when pasting it in bug reports (perhaps there could be a more confidential way to share as it turns out to be quite useful to find problems -- https://www.npmjs.com/package/cipherhub maybe).
Thanks @ericthegrey! That pointer helped me figure out that changing the fields to optional makes it possible to parse the change update. Also, a good point regarding the logging output, I've removed that for now and will look into other ways of sharing debug logs.
@Martin-bcd, would you mind testing the updated branch to see if this fixes the issue? You can use the same commands as in your message above. Thanks!