Unable to restore encrypted backup - Bad MAC
mike-edel opened this issue · 122 comments
- I have searched open and closed issues for duplicates
- I am submitting a bug report for existing functionality that does not work as intended
- I have read https://github.com/signalapp/Signal-Android/wiki/Submitting-useful-bug-reports
- This isn't a feature request or a discussion topic
Bug description
Unable to restore signal backup on new phone.
Similar to #7637 but I don't have xposed installed.
Signal stable closing without error after reading 11401 messages.
Signal beta giving error message about incorrect passphrase after reading 11401 messages.
I am certain the passphrase and the backup are correct as they are working fine using https://github.com/xeals/signal-back
Steps to reproduce
- copy signal backup to new phone
- open signal
- enter passphrase
- wait for messages to import
Actual result:
- Signal just closing (stable)
- Error message stating the passphrase is incorrect (beta).
Expected result: Backup is successfully restored.
Screenshots
beta error (in German due to phone language)
Device info
Device: Nokia 7 Plus Dual-Sim
Android version: 9
Kernel version: 4.4.146-perf+
Signal version:
Beta: 4.30.2
also tested on stable: 4.29.7
Link to debug log
Working on grabbing a logcat log but struggling with it. Will add through edit.
Logcat for version 4.29.7:
--------- beginning of crash
11-08 22:32:10.330 4472 6987 E AndroidRuntime: FATAL EXCEPTION: AsyncTask #3
11-08 22:32:10.330 4472 6987 E AndroidRuntime: Process: org.thoughtcrime.securesms, PID: 4472
11-08 22:32:10.330 4472 6987 E AndroidRuntime: java.lang.RuntimeException: An error occurred while executing doInBackground()
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at android.os.AsyncTask$3.done(AsyncTask.java:354)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at java.util.concurrent.FutureTask.run(FutureTask.java:271)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at java.lang.Thread.run(Thread.java:764)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: Caused by: java.lang.NegativeArraySizeException: -1884239691
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:310)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:394)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:386)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at android.os.AsyncTask$2.call(AsyncTask.java:333)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11-08 22:32:10.330 4472 6987 E AndroidRuntime: ... 4 more
logcat for version 4.30.2:
11-10 13:15:20.687 27588 31822 W RegistrationActivity: null
11-10 13:15:20.687 27588 31822 W RegistrationActivity: java.io.IOException: Bad MAC
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readAttachmentTo(FullBackupImporter.java:298)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.processAttachment(FullBackupImporter.java:140)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:87)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at android.os.AsyncTask$2.call(AsyncTask.java:333)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
11-10 13:15:20.687 27588 31822 W RegistrationActivity: at java.lang.Thread.run(Thread.java:764)
So basically now coming from #8347 it seems i evolved to this one here. With the latest 4.30.7 on Nexus 6P it finishes after 2 hours my backup and my Pixel3 - 4.30.7 Signal tells me that my password is wrong. Same problem as above. Tried https://github.com/xeals/signal-back and worked with the same file and same password. Since i reached from crashing Signal on Pixel with restoring backup, i think the next Level is succesfully restoring a backup. :D
Pixel 3 - https://debuglogs.org/8afd5e209b77295d13b983b38b06c71fd0cbe32f04e4521f5f5aad5b2f87a4d3
Nexus 6P - https://debuglogs.org/703329cfc7d727fab8c4cc4087552e37cde3d45abd8f88428ef92ea91d3ee583
Same here. Running LineageOS 16.
Interestingly, with the wrong password specified, Signal will immediately claim that the password is incorrect.
With the right password, it reads around 2000 messages and then suddenly has a change of mind and claims that the password is incorrect after all. Seems legit.
***I tried both 4.26.2 and 4.30.8.
4.26.2:
W RegistrationActivity: at com.google.protobuf.CodedInputStream.pushLimit(CodedInputStream.java:651)
W RegistrationActivity: at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:307)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame.<init>(BackupProtos.java:4637)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame.<init>(BackupProtos.java:4514)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame$1.parsePartialFrom(BackupProtos.java:4675)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame$1.parsePartialFrom(BackupProtos.java:4670)
W RegistrationActivity: at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
W RegistrationActivity: at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
W RegistrationActivity: at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
W RegistrationActivity: at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
W RegistrationActivity: at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.BackupProtos$BackupFrame.parseFrom(BackupProtos.java:4937)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:328)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:394)
W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:386)
W RegistrationActivity: at android.os.AsyncTask$2.call(AsyncTask.java:333)
W RegistrationActivity: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
RegistrationActivity: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W RegistrationActivity: at java.lang.Thread.run(Thread.java:764)
4.30.8:
W RegistrationActivity: null
W RegistrationActivity: java.io.IOException: Bad MAC
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:320)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
W RegistrationActivity: at android.os.AsyncTask$2.call(AsyncTask.java:333)
W RegistrationActivity: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W RegistrationActivity: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
RegistrationActivity: at java.lang.Thread.run(Thread.java:764)
I have restored backups (not this specific file) in the past a couple of times with no issues. The only change is that when it worked it was with Android 7.x and now I'm on 9.x. Probably not relevant though.
Clearly, the backup was a) generated by signal itself and b) in good health since signal-back can properly decode it.
9.x. Probably not relevant though.
But maybe this is the relevant part
9.x. Probably not relevant though.
But maybe this is the relevant part
Indeed, seems to be a common thing...
Not valid anymore! I had different Signal versions! Sorry for any confusion.
Same problem here with Android 8.1
Signal crashes after it reads the 4000+ Message. So perhaps it is not the same Issue? I cropped my conversations and deleted all media inside signal conversations, but that did not help. I also can open the backup with signal-back (only "umlaute" (ÄÜÖ) are displayed wrong).
^^ There goes the 9.x theory.
By the way, I did not see this issue restoring on LineageOS 15.1 (Android 8.1). Different backup file though (another account).
Actually, it may be possible that the backup was not restored. This was a few weeks ago and my memory is flimsy. Sorry about the spam, not possible to edit comments anymore it seems..
I had different Signal versions! Sorry for any confusion.
So perhaps still a 9/pie issue?
Also experiencing this migrating from Nexus 6 Android 7.1.1 to Moto X4 Android 8.1.0
Decrypts messages (as it is counting the correct number to be restored) Then when it fails it throws Incorrect password. Log cat shows the Bad Mac error as well.
Has anyone found a work around?
Stock Roms (Nexus 6, Moto X4 Android One edition)
Signal 4.30.8 on both as well
Had the chance to day to try on Android 8.1 AOSP.
I zygote : Deoptimizing void net.sqlcipher.database.SQLiteClosable.releaseReference() due to JIT inline cache
W RegistrationActivity: null
W RegistrationActivity: java.io.IOException: Bad MAC
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:320)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:252)
W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
W RegistrationActivity: at android.os.AsyncTask$2.call(AsyncTask.java:333)
W RegistrationActivity: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W RegistrationActivity: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
W RegistrationActivity: at java.lang.Thread.run(Thread.java:764)
In one swift strike, it seems that we can rule out the culprit being Android 9 or LineageOS.
Perhaps @moxie0 has something to add at this stage?
This is the very same backup file from last month by the way, which signal-back can perfectly read..
Hi,
I also have this issue coming from an Android 7.1.2 and going to an Android 9.0.2
As far as I understand, this seems to be related to #8327
@greyson-signal @elkhadiy can maybe help us as they were the ones involved in #8327
I am using the 4.31.6 version which seems to be corresponding to the master version.
Still I have the issue.
Hi,
Just skimmed through this thread, I'll be back later on to help.
What I can say for now is that before 4.30.2, when restoring a backup, Signal was happily decrypting all messages without checking their message authentication code (MAC). #8327 fixes this check.
Now it seems we are dealing with some corrupt data in the backup. I'll hack together a tool to try and find the offending frames. (I'll keep you guys posted)
EDIT(2018-12-16):
So I'm a little puzzled by the fact that OP can decrypt the backup file with signal-back while he has a Bad MAC error. Are you sure the whole file is decrypted and that it didn't just decrypt the good frames? Could you use my tool : https://github.com/elkhadiy/debruitage ? I made sure to make it explicitly report all bad frames it encounters during the process.
EDIT(2018-12-18): Removed superfluous information irrelevant to the issue at hand.
I ran your tool and got the following exception:
Traceback (most recent call last):
File "/home/royalvein/bin/miniconda/envs/atom/bin/signal-bkp-decrypt", line 11, in <module>
load_entry_point('signal-backup-manager==0.1.dev1', 'console_scripts', 'signal-bkp-decrypt')()
File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/signal_backup_manager/cli.py", line 23, in run
bkp = SignalBackup(args.backup_file, args.passphrase)
File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/signal_backup_manager/signal_backup.py", line 22, in __init__
+ datetime.now().strftime("%Y-%m-%d--%H-%M-%S")
File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/fs/osfs.py", line 311, in makedir
_path = self.validatepath(path)
File "/home/royalvein/bin/miniconda/envs/atom/lib/python2.7/site-packages/fs/base.py", line 1429, in validatepath
else "paths must be str (not bytes)"
TypeError: paths must be unicode (not str)
Exception AttributeError: "SignalBackup instance has no attribute 'preference_file'" in <bound method SignalBackup.__del__ of <signal_backup_manager.signal_backup.SignalBackup instance at 0x7f77295e3200>> ignoredI do not know if it is the expected behavior...
Basically, I clone your project, and follow the readme (pip install...).
However, I did not try signal-back before so i did it at the same time and I was not able to extract my messages and got:
error: failed to open backup file: failed to read headerLengthBytes: EOF
Maybe my issue is not the same after all even if the behavior seems similar (bad mac error + count of messages is good) to @spospartan104
So I try again to import my message and get the exact exception which is not the same as @sarevok-anchev (in term of the line numbers but maybe the version differs)
Here are the logs:
W RegistrationActivity: null
12-19 23:45:39.169 4632 7965 W RegistrationActivity: java.io.IOException: Bad MAC
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readAttachmentTo(FullBackupImporter.java:298)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.processAttachment(FullBackupImporter.java:140)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:87)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:396)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:388)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at android.os.AsyncTask$2.call(AsyncTask.java:333)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
12-19 23:45:39.169 4632 7965 W RegistrationActivity: at java.lang.Thread.run(Thread.java:764)Oh my bad, that's because it's written in python3 so you can go ahead and pip uninstall signal-backup-manager then pip3 install .
Though signal-back's error message is weird, what's the first 4 bytes of your backup file?
~$ echo $(( 16#$(xxd -p -l 4 signal-yyyy-mm-dd-hh-mm-ss.backup) ))Well... My bads...
The file I get to run the command was corrupted.
I redownload it and now there is no error with signal-back and your tool.
However there still the Bad Mac exception on android.
echo $(( 16#$(xxd -p -l 4 signal-yyyy-mm-dd-hh-mm-ss.backup) ))gives me 54 with the non corrupted file.
Are you sure Signal tries to restore from the file you checked?
For example you only have that one in /sdcard/Signal/Backups/?
Yep I try with 2 different backups but each time, only one backup was available in this folder.
Hi,
Well today, I choose to reset my phone because I mess with it to do some tests with multiple apps.
And I choose to retry the installatio of Signal and the import of my backup and everything worked fine.
I don't know what was the cause of this issue but it seems resolved for me after the reset.
Anyway thanks for the help. Signal is really a cool app and I missed it !
@elkhadiy I tried your tool and there's an error:
Bad frame @ file offset 645847
Raw Frame data:
b'\xd1\x0e\xd7\xc8\xcc\t\xd0/\x06/\xe5#\xb6\xcbo2:\xab\xfc\xa4]\x1e\xc3\x8e\xf4!C\xdd"\xd1\x15\xd3<\xa6\xea\xca\xeb\x9c$\xc5\n\x01\x12\x83v\x0eC\x0c\x14\xccq\xba\x96#\x123d\x98\xe4\x82\xb4m\xd3/\xa1\x15\xa2\xef1\xa4\r\xdd\x9d\xf3\x17$\x87\x0c\x9d\x11\n\xf3A\x16_\xf7i\x81\xf2NKQ\xd5Zd<\x05\t.k\x03\xa5X\xb5gN\x04\x0cF\xf8\xb2\xbd\x92\xed(\xe5.\xb7@\xad\xdcM\xbe\xd5*m5\xbfeM}\x07\x10t>\xada\x1b\x8b\xe2\x15\x91oE\xbb\xf8\xa9\xa8E8z\xc4\xca\x87{\x82X_di\xcf\xb9\xad\xec\xb9=\xf13\x11\xfc<\xf9i;\xd8sx\x83\xe3\x8b:r6\xadX\x1d\xac2\xed@\xba\xbc\xc0`\xb0\x9d\x07\xc7\xab\xe6j\xf0#`\x9e\x10\x18\x12\xcc\xf8)\xb5\xdeC\xba'
Attempting to build frame obj:
Traceback (most recent call last):
File "/home/user/.local/bin/signal-bkp-decrypt", line 11, in <module>
load_entry_point('signal-backup-manager==0.1.dev1', 'console_scripts', 'signal-bkp-decrypt')()
File "/home/user/.local/lib/python3.6/site-packages/signal_backup_manager/cli.py", line 23, in run
bkp = SignalBackup(args.backup_file, args.passphrase)
File "/home/user/.local/lib/python3.6/site-packages/signal_backup_manager/signal_backup.py", line 50, in __init__
for frame in self.__get_backup_frames():
File "/home/user/.local/lib/python3.6/site-packages/signal_backup_manager/signal_backup.py", line 130, in __get_backup_frames
frame.ParseFromString(plaintext)
google.protobuf.message.DecodeError: Error parsing message
Note that:
-
This is the only backup file in /sdcard/Signal/Backups, so I'm 100% sure it's the one Signal is attempting to use.
-
Signal-back can decrypt the backup: signal-back check $filename result is
Backup looks okay from here. -
Earlier you said:
What I can say for now is that before 4.30.2, when restoring a backup, Signal was happily decrypting all messages without checking their message authentication code (MAC). #8327 fixes this check... however I've attempted to restore with 4.29.7 it's not possible either. I do remember having restored this backup with previous versions, but a) I don't remember which version and b) of course the backup file has changed since then, and therein might lie the problem.
I must say it's a little bit frustrating that the option to set a very long passphrase and encrypt the app data was removed in previous versions, I never lost a single backup that way over the years, now I'm forced to relay on the signal backup functionality, which apparently in certain unknown circumstances creates corrupted backups, and it's already the second time I'm losing data.
Personally I brought more than 100 people into Signal over the years, and now I can't reach any of them for more than a month because of this bug.
A bit offtopic, but I needed to vent - I had a good grip on backup flow, now need to rely on a dumbed down version, and it's the second time that it screws my data up. Not good.
Anyway..
Let's look towards a possible solution.
Can you offer some guidance on what the best next steps are from this position?
I don't want to lose the conversation history yet again, and since there's roughly a month and a half of undelivered messages now, I would like to keep the same encryption keys so that they get delivered (eventually).
The error must be in some corrupted attachment. signal-back has (on quick glance) all of the messages. It would be way less catastrophic to lose some attachments rather than lose the whole conversation history.
Nevertheless, signal-back can recover a lot of images and audio from this supposedly corrupted backup. Maybe even all of them - I wouldn't know.
Is it possible to make signal ignore a corrupted attachment rather than abort the backup in an all-or-nothing fashion?
Perhaps even have an option to ignore such errors altogether (at the users' peril) ?
How about extending your tool to ignore the errors and discard the file/message in question, and outputting a new backup file minus the "bad frames" ? That could be a fast solution to get signal going quickly again.
What do you think @elkhadiy?
Oh yeah sorry I derped in my tool, forgot to catch the potential frame construction exception.
Signal-back can decrypt the backup: signal-back check $filename result is Backup looks okay from here.
Hmmm weird... I'll try and see what he does (but I'm not that good with golang).
however I've attempted to restore with 4.29.7 it's not possible either.
Yeah pretty sure you won't restore from this backup with the current state of affairs since at least this frame seems to be pretty borked, if protobuf can't parse it...
Is it possible to make signal ignore a corrupted attachment rather than abort the backup in an all-or-nothing fashion?
How about extending your tool to ignore the errors and discard the file/message in question, and outputting a new backup file minus the "bad frames" ? That could be a fast solution to get signal going quickly again.
Yep, was thinking about proposing this in the app's backup restore process. Something flexible that would report bad messages or attachments and restores anyway...
I'll put together something that'll analyse the database entries and reports what is most likely missing (sms/mms entry or attachment and maybe even its filename I think)
I was also planning on writing a module that rebuilds the backup using the same passphrase anyway for a friend of mine that wants to import his Facebook messenger history. Pretty sure we can fix your backup file!
I'll get back to you hopefully very soon.
Beautiful, thank you.
@elkhadiy Thanks for the tool - was struggling a bit to get it working on Windows. Been a while since I ran stuff in a shell. Getting some odd errors:
Bad frame @ file offset 76 Raw Frame data: b'\xf3EaU' Attempting to build frame obj:
Full console output:
signal-bkp-decrypt.exe -b c:/signal-2018-11-08-22-04-02.backup -p '040871752236135417905193908720' Bad frame @ file offset 76 Raw Frame data: b'\xf3EaU' Attempting to build frame obj: Traceback (most recent call last): File "C:\Users\mikee\AppData\Local\Programs\Python\Python37-32\Scripts\signal-bkp-decrypt-script.py", line 11, in <module> load_entry_point('signal-backup-manager==0.1.dev1', 'console_scripts', 'signal-bkp-decrypt')() File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\signal_backup_manager\cli.py", line 23, in run bkp = SignalBackup(args.backup_file, args.passphrase) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\signal_backup_manager\signal_backup.py", line 50, in __init__ for frame in self.__get_backup_frames(): File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\signal_backup_manager\signal_backup.py", line 130, in __get_backup_frames frame.ParseFromString(plaintext) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\message.py", line 185, in ParseFromString self.MergeFromString(serialized) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1083, in MergeFromString if self._InternalParse(serialized, 0, length) != length: File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 1109, in InternalParse new_pos = local_SkipField(buffer, new_pos, end, tag_bytes) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 850, in SkipField return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 799, in _SkipGroup new_pos = SkipField(buffer, pos, end, tag_bytes) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 850, in SkipField return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end) File "c:\users\mikee\appdata\local\programs\python\python37-32\lib\site-packages\google\protobuf\internal\decoder.py", line 782, in _SkipFixed64 raise _DecodeError('Truncated message.') google.protobuf.message.DecodeError: Truncated message.
Don't know what to make of it to be honest. And no idea whether signal-back does actually decrypt the whole thing. I ended up getting a huge xml file though.
Any updates @elkhadiy ?
i'm also seeing this on a friend's phone, trying to restore a fresh backup from an old phone on a new one.
@elkhadiy your tool is great, and indeed it complains about a corrupted frame. could it be extended to re-encrypt the backup to a new file with all corrupted frames dropped?
I've just seen this on my phone; I was unable to send (#8602) and figured a reinstall would probably do the trick, but I'm unable to restore my messages. Fairly chunky backup at 60,000 messages. Can't get debruitage to run on my desktop as Ubuntu doesn't have the fts5 extension for sqlite packaged, but will try again on Arch later. Having Signal ignore (with some kind of message) corrupt MAC frames would be the ideal outcome, I think - losing 60,000 messages for one glitch isn't fabulous.
Just wanted to let you guys know, I'm working on this. I've been able to decrypt (and interpret) the backup data for a long time and just started working on reencryption last week and making good progress.
I think a lot of the past messages shouldn't be in this github issue as they do not relate to any bug in Signal (supporting corrupted backups would be a feature request). The only possible bug here is that signal pops up a wrong password error when a bad Mac is encountered, while I think a 'file corrupted' popup would be way more likely if it happens any time after the first frame is decoded. So, if I have something to test for you, I will create a thread on the forum and post a link here. Note it might take a long time (think months) , as I sometimes don't have spare time for an extended period.
Also, I do not know how Signal handles backups with frames missing, so there are no guarantees this will work. Frames are not necessarily independent things (attachment data belongs to an entry in the 'part' database, which belongs to an entry in the 'mms' database. The restoration might for all I know still fail if one of those frames is missing. Also, random corruption in a random location might result in anything, that is, if you're unlucky, a single bit flipped might make not just one frame, but all following frames unreadable.
Anyway, I didn't intend for this message to get this long. Just wanted to let you know, a tool is coming though it might be a while.
The only possible bug here is that signal pops up a wrong password error when a bad Mac is encountered, while I think a 'file corrupted' popup would be way more likely if it happens any time after the first frame is decoded.
we were able to reliably reproduce 'corrupted' files. no matter how often we made new backups on the one phone, they always turn out to be corrupted on the other one. i wouldn't rule out the possibility that signal's backup routine has a hidden issue leading to corrupted frames in backups which triggered it, and the restore problems are merely a symptom.
I'd +1 that suggestion, @unDocUMeantIt - my phone has dozens of GB free, backups were allowed to complete fully onto internal storage with no sleep/screenlock, and restore attempts likewise. I can't think of any potential source for corruption of the backup other than in creation by Signal.
If the error is caused because the contents themselves are also being checked during restore, but in normal operation such corrupted contents are tolerated, then the behaviour should be modified to catch what would cause corruption at "point of entry" to storage and abort then, and to permit restore behaviour to ignore those corrupt frames on restoration.
Well, first of all, I should indeed not have been so self assured stating there is no bug here. I am not a Signal developer and I am not 100% sure about anything.
However, having said that, being very familiar with the backup file format, and having read the code, I still find it very unlikely that a bug is present. You should realize the MAC is just a hash of the already encrypted bytes of data. During the backup creation process, data is serialized into an array of bytes, these bytes are then encrypted to produce another, different array of bytes. Then a hash is calculated from these bytes (and, indirectly, your passphrase), this hash is the MAC. Then, the encrypted bytes and the hash are written to the file.
Notice that serializing the data could go wrong, but it does not matter (I don't know if Signal even checks for this). Encrypting the data could go wrong as well, creating an impossibble-to-decrypt array of random bytes. This also does not matter. The MAC is calculated on the (bad, uninterpretable, random) array of bytes without regards to what these bytes represent.
During the backup importing process. First the array of encrypted bytes is read into memory, the same hash is calculated from them, and compared to the stored MAC. No decrypting is taking place, the bytes are in no way interpreted at this point and it is not known wether the bytes can be decrypted or would turn into a valid signal backup frame if they were. If the hash does not match the stored hash, you get the "Bad MAC" exception in the log and an 'incorrect passphrase' message pops up (erroneously). But really the MAC can not show if the bytes are 'correct' on the receiving phone, only that they are the same. Having a bad mac shows the bytes have not arrived unaltered on the new phone. This means one of three things:
- The MAC is calculated wrong (and differently on the two phones). This would mean a bug in Java's crypto package. Unlikely...
- The passphrase is wrong. Very unlikely if it has managed to successfully calculate the MAC of even one frame before.
- The bytes have been changed. ... likely? I think this would mean one of the phones or inter-transfer storage has bad blocks of memory or some connection issues during transferring the backup file.
Again, I could be wrong about this.
If anyone still has a phone which reliably reproduces bad backups, try this:
- Create a backup, let it finish. Rename the backup, do NOT delete it.
- Immediately create another backup. Make sure the backup contents do not change in between (do not receive any messages). Both backups exist simultaneously on the same device, so they cannot occupy the same (possibly bad) flash memory.
- Calculate checksums (for example md5sum) of both files on the phone, so that after transferring it can be checked that the copies arrive identical to the original.
- Verify that the exact same frame gives the 'Bad MAC' exception from both backup files.
I think this would make it much more likely that Signal is the problem. Step 4 might be a little tricky at this moment, by my (forthcoming) program would make it easy.
I have by now, by the way, been able to read a backup, actually edit the contents of one of the messages and reencrypt with a new passphrase. I then imported this new backup on an actual phone with a normal signal installation successfully. I expect to be able to post a first version dropping bad frames later today (or at least this week). To compile the program you will need a recent c++ compiler (I use gcc version 8.2.1 20181127), and libcryptopp.
I have posted a thread with the tool in its current shape here: https://community.signalusers.org/t/tool-to-re-encrypt-signal-backup-optionally-changing-password-or-dropping-bad-frames/6497
I hope it can help some of you.
I can confirm that this "Bad MAC" bug still exists. I tried to migrate my Signal data from LineageOS 14.1 (Android 7.1, Signal version 4.35.3) to LineageOS 16.0 (Android 9.0, Signal version 4.35.3).
I went through the first three steps of bepaald's suggested test (#8355 (comment)). Both identical backups I created were transferred correctly to the new phone (sha256sums were the same) and both failed with the "Bad MAC" error in the log when I tried to import them.
I will now try bepaald's script to remove bad frames...
I'm seeing this same problem while migrating data to a new phone.
Exporting from: Signal v4.39.4 on Android 8.1.0 (Pixel 1)
Importing to: Signal v4.39.4 on Android 9 (Pixel 3a)
The backup file is about 1GB. It gets about 1900 records into the import, then give me the "Unable to restore encrypted backup" error message. I hooked up adb logcat and got this:
05-22 19:41:25.948 564 609 W SurfaceFlinger: Attempting to set client state on removed layer: Dim Layer for - Task=100#0
05-22 19:41:25.948 564 609 W SurfaceFlinger: Attempting to destroy on removed layer: Dim Layer for - Task=100#0
05-22 19:41:26.071 766 1033 I CHRE : @ 24149.145: [AR_CHRE] still: 92
05-22 19:41:26.599 24220 24375 V Cursor : Filling cursor window with start position:0 required position:0
05-22 19:41:26.634 24220 24375 I FullBackupImporter: Ignoring import for statement: CREATE TABLE sqlite_sequence(name,seq)
05-22 19:41:29.272 766 1033 I CHRE : @ 24152.344: [AR_CHRE] still: 87
05-22 19:41:32.468 766 1033 I CHRE : @ 24155.543: [AR_CHRE] still: 100
05-22 19:41:34.190 24220 24375 W RegistrationActivity: null
05-22 19:41:34.190 24220 24375 W RegistrationActivity: java.io.IOException: Bad MAC
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readAttachmentTo(FullBackupImporter.java:299)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.processAttachment(FullBackupImporter.java:141)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:87)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:376)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:369)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at android.os.AsyncTask$2.call(AsyncTask.java:333)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
05-22 19:41:34.190 24220 24375 W RegistrationActivity: at java.lang.Thread.run(Thread.java:764)
05-22 19:41:35.668 766 1033 I CHRE : @ 24158.742: [AR_CHRE] still: 100
05-22 19:41:37.729 1167 2703 W NotificationService: Toast already killed. pkg=org.thoughtcrime.securesms callback=android.app.ITransientNotification$Stub$Proxy@4170b12
I've tried re-doing the export with the same results. It seems to be choking on that one attachment for whatever reason.
I have to disagree with @bepaald about this not being a bug. If Signal can generate backups with invalid attachment MACs, seems to me there's a bug in the way the MAC is computed or the formatted output is written.
I wish I could provide a reliable way to reproduce this without supplying my entire 1GB backup file and all my message & attachments. I think I'm going to see if I can get Android Studio set up and reproduce this against my data in a debugger.
Did a little hacking. Breakpointed it and caught the error in a debugger. It is indeed a failed MAC match:
It seems mildly interesting that the plaintext of the attachment (the output of cipher.doFinal()) is a zero-byte array (lines 282 and 285) despite 13693646 bytes having been fed to the cipher. Any crypto experts on this thread? What does it mean if you try to decrypt AES CTR mode and get zero-byte plaintext output?
Makes me think something has gone wrong either with the encryption of the attachment, or with frame alignment while streaming in the backup file.
Just for fun, I tried disabling the MAC check by commenting out the line that throws the exception and running that import, and it ran out of RAM:
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: java.lang.RuntimeException: An error occurred while executing doInBackground()
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at android.os.AsyncTask$3.done(AsyncTask.java:354)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at java.util.concurrent.FutureTask.run(FutureTask.java:271)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at java.lang.Thread.run(Thread.java:764)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: Caused by: java.lang.OutOfMemoryError: Failed to allocate a 1802744496 byte allocation with 4572351 free bytes and 507MB until OOM, max allowed footprint 9144703, growth limit 536870912
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:314)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at org.thoughtcrime.securesms.backup.FullBackupImporter$BackupRecordInputStream.readFrame(FullBackupImporter.java:256)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at org.thoughtcrime.securesms.backup.FullBackupImporter.importFile(FullBackupImporter.java:81)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:376)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at org.thoughtcrime.securesms.RegistrationActivity$2.doInBackground(RegistrationActivity.java:369)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at android.os.AsyncTask$2.call(AsyncTask.java:333)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
05-22 23:57:15.721 5531 5587 E UncaughtExceptionLogger: ... 4 more
Also, I've verified that the MD5 sums of the backup files on the source and target phones match, so I'm pretty sure it's not corruption due to failed copying, although it's possible that the source phone could have failed to write the data correctly in the first place.
I'm still intrigued by this issue. Do you happen to have enough free space on your device to get two backup files on there at the same time? I would still like to see the problem occur at the exact same place in two different backups (as described here).
I have been rewriting my program, which scans for bad frames, to actually build a database so it will be easier to see exactly what attachment (and what message) causes the problem. Unfortunately, I can't get it to compile under Windows (that is, I do not have Windows and cross-compiling isn't working). But, if you feel comfortable using Linux, I will happily give you the sources. The program might be able to tell you exactly what message the attachment belongs to and will try to dump the attachment data (corrupted or not) so you can inspect the attachment both in Signal as well as what is in the backup. Let me know if you want to try. Also, as a suboptimal workaround, if you know what message causes the problem, you could try to get a working backup by deleting it.
It seems mildly interesting that the plaintext of the attachment (the output of cipher.doFinal()) is a zero-byte array (lines 282 and 285) despite 13693646 bytes having been fed to the cipher. Any crypto experts on this thread? What does it mean if you try to decrypt AES CTR mode and get zero-byte plaintext output?
I am definitely NOT a crypto-expert, but I don't think this is an issue. The doFinal() message is really mainly to reset the Cipher, and yes, it might return some data if there was still some left in the buffer, but I think that would actually be the unusual case.
Note: the following is even more speculation then everything else I'm saying:
Just for fun, I tried disabling the MAC check by commenting out the line that throws the exception and running that import, and it ran out of RAM
I think this is interesting as well. The backup importer doesn't just read data until it gets some end marker, instead every frame is preceded by 4 bytes giving the length of the next frame, which is then read. So, even if a frame was bad - the encryption gives bad output and the MAC is calculated or written incorrectly - the next frame should be all good again. The exception above shows it's trying to allocate almost 1.7 Gb (that is more than the entire backup file!) which might mean it is reading a corrupted frame size. Note these four bytes framesize are not part of the frame itself and are not encrypted, just a plain 32bit integer in between each frame. So this might mean more data is corrupted in the backup file (and not because of encryption). If this is the case I would have expected the exception to be thrown at line 311, did the code move down a few lines in your edits?
[...] although it's possible that the source phone could have failed to write the data correctly in the first place.
This is still the first thing I would like to see ruled out. The input to the encryption functions varies greatly, just toggling the backup preference changes the keys used for encryption, and thus the data that is hashed to get the MAC (the MAC is calculated from the encrypted data). In fact, toggling the preference is probably not even necessary as there is random key material generated when a new backup is created. The chance that a second backup has the same bad MAC seems really low, unless of course the bug is very common and easy to run into, but then many more people should have this issue. Anyway, I'm not saying there is definitely no bug, I just still think the more likely explanation should be ruled out first.
@bepaald Oh, yeah. You're correct about doFinal(). I misread that the first time. Plaintext is getting iteratively written to the output stream after successive calls to cipher.update().
I'd love to get ahold of your verification code. I'm running Linux as well. I do have multiple exports I can compare, taken at slightly different points in time.
I also my backup through xeals/signal-back extract, and the attachment with the same ID that fails MAC check in the debugger "decrypts" to 14MB of garbage, which supports the corrupted-on-export theory. All the prior attachments extract just fine. signal-back check doesn't find any problems, but I don't think it's checking attachment frames at all.
I would have expected the exception to be thrown at line 311, did the code move down a few lines in your edits?
Yeah, I added a few lines for debugging & then commented them out.
@deleted Here you go: https://send.firefox.com/download/9150b1f452b16fc8/#DHCNlAGrBkVpcUqtWONO9A. See newer code in a later post.
There is a little buildscript that works for me, but might need some editing depending on your library paths and such. It should at least give you an idea how to compile it. Make sure you have a recent version of g++, and have (development packages of) crypto++ and sqlite3.
Also, this program is slightly edited, custom made for you, it does not actually have all the options it says when run without arguments. Just run ./badmacinfo name-of-backup-file.backup [30digitspassphrase] and it will try to do its thing. I hope it will be useful somehow.
Let me know if you need help with this. If you have any bright ideas that can be tested with the some adjustments to the program let me know (or code them yourself of course).
If you have a second phone, it might be interesting to forward the attachment (possibly the only unchanging bytes in your multiple broken backups) to that phone and see if its backups also turn bad...
@bepaald Thanks for the code. Unfortunately I haven't been able to build it.
Furthest I've been able to get is in a debian:sid container, but it still doesn't seem to distribute an up-to-date version of libcrypto++:
BUILDING (2/22): gcc -std=c++2a -c -Wall -Wextra -Wshadow -Wold-style-cast -Woverloaded-virtual -pedantic -fomit-frame-pointer -O3 -march=native -o "cryptbase/o/getbackupkey.o" "cryptbase/getbackupkey.cc"
cryptbase/getbackupkey.cc: In member function 'bool CryptBase::getBackupKey(const string&)':
cryptbase/getbackupkey.cc:27:13: error: 'byte' is not a member of 'CryptoPP'
CryptoPP::byte digest[CryptoPP::SHA512::DIGESTSIZE];
This page seems to suggest that the CryptoPP:byte type was added in libcrypto++ 6.0.
What distro & version are you using?
@deleted O wow, I wouldn't have thought the 'unstable' version of debian would come with an almost three year old version of cryptopp. I'm running arch with cryptopp 8.2.0, I see that version is also available in debian 'experimental', is it possible to install that one? Alternatively, you could try to just remove the CryptoPP:: from the all the occurrences of CryptoPP::byte, as you probably already learned from the page you linked, byte was in the global namespace in the earlier versions of cryptopp. But it might lead to a clash with the byte added in c++ (though I do not have a using directive for std::) and you might run into other problems with the old version of cryptopp, I don't think I have ever written programs with version prior to 6, I don't know if everything will be backwards compatible.
Got it to work in an Arch container.
For anyone following along from home, here's the Dockerfile I ended up with:
FROM archlinux/base
RUN pacman --noconfirm -Sy gcc crypto++ sqlite3 libstdc++5
ADD . /src
WORKDIR /src
RUN bash BUILDSCRIPT.sh
I also had to add -lstdc++ to the end of the last line of the build script to get it to link properly.
Thanks for this. The info it dumps on MAC failure is really helpful.
Two different exports indeed fail on the same attachment file. I'm going to try deleting that and redoing the export.
@bepaald Update:
I deleted the entire conversation with the bad attachment in it. That allowed my to export an uncorrupted backup, so I'm likely going to walk away from this problem for now.
If you're still intrigued, I've posted to output of your tool:
https://github.com/deleted/badmacinfo-output
That has the stdout from the tool (with phone numbers scrubbed) and the binary of the decrypted attachment. It's an animated .gif that seems to display fine, but has some artifacts near the end that might be corrupted data.
Thanks for your help! The badmacinfo tool was really useful for me. I'd encourage you to publish the source here on github, if you're so inclined. It may help others.
@deleted Thanks for following up! In my testing, I just changed 10 bytes in one of my backups and happened to catch an animated gif as well. I also just saw a few artifacts in one of the last frames, so that seems about right. I can't seem to open the file you uploaded at all by the way, no program I tried recognizes it as a valid image (it doesn't even start with a gif signature).
I don't think there is very much I can do without actually having a broken backup myself, and even then it's probably a difficult problem to definitively solve. Some questions that could possibly have been interesting to get answers to (since you deleted the conversation, you probably can't answer any of these anymore, and that's ok, but I'm going to ask them just in case, or for the next person that comes to this thread with this problem):
Did you also try deleting just the one single message, and did that not work? And also, did you view the .gif from within your signal conversation before deleting it, did it look good from there? Did you happen to save the image from your conversation, and did that work and was it uncorrupted?
Anyway, I'm glad you have been able to get a working backup and use your new phone with (most of) your messages transferred.
EDIT
Thanks for your help! The badmacinfo tool was really useful for me. I'd encourage you to publish the source here on github, if you're so inclined. It may help others.
Yeah, I plan to release some code sometime in the future, it's just not there yet (as far as the main features I had planned, mainly editing the backup). Also, I have an exporter that is able to reencrypt a parsed backupfile, and a (mostly) working GUI for viewing the backup files on PC (should look almost exactly like on your phone's screen. But - and don't ask me how this happened - they're three separate programs and I feel like I should probably merge them into one giant signal-backup-tool. Now if only I had all the free time in the world...
In the meantime I hope people just find this thread or the one in the community forums where I also managed to fix someones backup with an older version of this tool.
@bepaald I had the same problem so I will actually test the viewing gif, deleting single message, etc in this. Will report back within the week I hope.
typical Signal App bad vibes from programmers :(
@bepaald So I grabbed the tool you posted above with the firefox send link but am not receiving the same output style as 'deleted' Not sure if it's because of the custom style or not
spospartan104@:~/bin/signal-debugging/badmacinfo
$./badmacinfo Signal/Backups/signal-2019-04-14-04-09-14.backup $ (cat backupcode) newfile
Reading backup file...WARNING: Bad MAC in frame: theirMac: (hex:) ce d5 9e a3 9f 39 53 e9 31 87
ourMac: (hex:) f6 49 63 4e f8 74 e6 22 5b de 35 b0 23 9e 25 93 53 f8 25 43 0b 44 49 12 00 71 15 d5 b7 4d bf d0
terminate called after throwing an instance of 'CryptoPP::InvalidKeyLength'
what(): AES/CTR: 0 is not a valid key length
Aborted (core dumped)
@spospartan104 I'm sorry, I don't know exactly how that can happen. Does it count the frames (and the percentage) up before this happens, or is this the very first thing it prints? Is it possible your backup is corrupted from the very first frame? What happens if you try to restore this backup in Signal, does it count up the frames and then suddenly stop, or does it fail immediately?
Of course it is also possible I have a bug in my program. In fact, there certainly is, it should not try to create an AES/CTR object with a 0 key size, it should fail earlier if the key has length zero (but I don't understand how it gets in that situation). I have a new version, which will not fix your problem, but might tell us something about why it fails. Maybe you could try this one https://send.firefox.com/download/a004f35d6b9b77a6/#lGciKr0mQpXxdNSZTmEodQ?
@spospartan104 Any luck with the code I posted earlier?
@bepaald sorry ran out of time the other day. Trying to see if I can build this now. I don't see a Make file so gotta remember my ol' gcc
EDIT: If you can provide your build steps I'd apprecaite it
@bepaald thanks for sharing your code, is there any chance you could make it available on github, even in an incomplete state? The send.firefox links you share have a short expiration time.
I read in another thread that you're new to adding code to github, this link might be helpful and I'm happy to give pointers if you like. https://help.github.com/en/articles/adding-an-existing-project-to-github-using-the-command-line
@bepaald Do you have time to upload the build steps?
`$g++ -std=c++17 main.cc
/usr/bin/ld: /tmp/cc3nxgid.o: in function 'main':
main.cc:(.text+0x436): undefined reference to
'SignalBackup::SignalBackup(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/usr/bin/ld: /tmp/cc3nxgid.o: in function 'SqliteDB::~SqliteDB()':
main.cc:(.text._ZN8SqliteDBD2Ev[_ZN8SqliteDBD5Ev]+0x23): undefined reference to 'sqlite3_close'
/usr/bin/ld: /tmp/cc3nxgid.o: in function 'FileDecryptor::~FileDecryptor()':
main.cc:(.text._ZN13FileDecryptorD2Ev[_ZN13FileDecryptorD5Ev]+0xf): undefined reference to 'vtable for FileDecryptor'
collect2: error: ld returned 1 exit status
` Current error
@spospartan104 Sorry for the late reply, it can get very busy for me sometimes. You should be able to compile the program with g++ -std=c++2a -Wall -Wextra -pedantic -O3 -march=native */*.cc main.cc -lcryptopp -lsqlite3 -o ./badmacinfo. Make sure you have sqlite3 and crypto++ installed. Depending on your setup, you might also need to add some -L flags to tell the linker where to look for certain libraries.
By the way, I thought you had already built the program earlier (when it didn't work), how did you run it? I f you didn't build it, but just used someone else's executable, that could very well be why it failed before.
@samizdis Yes I realize it's annoying. I do plan to make the code permanently available sometime in the future. It's just not in a presentable state right now, it's very quickly and heavily edited to provide a hint of functionality to the people in this thread. And it lacks many basic but important things (proper argument parsing and error checking). If in the meantime you (or anyone else) needs a re-upload, just let me know.
@bepaald a re-upload would be great, thanks, I'm trying to work out which of my messages has a bad MAC, hopefully if I delete that message then I'll be able to create a restoreable backup 🤞
Thanks @bepaald .
Just to echo what @spospartan104 said, code can definitely be work-in-progress and still hosted on GitHub.
@bepaald Got it working. And I have the frame info output. Now I need to hunt down what it was.
I do see "pending_push: 1" So my guess is it's an attachment that failed to send? if that's what it means.
Quick Edit. Think I found it, giving it a go. This is good stuff. Thank you again
Just to feed back that @bepaald 's code found two messages in my backups which had attachments with bad MACs. When I looked at these messages in the Signal app they'd both failed to either send or receive. After deleting these messages in the app I created a new backup which I was able to restore from on a different phone.
@bepaald your code is definitely useful enough and user-friendly enough to go on github, I'm sure other people would also find it useful even if you still plan on doing more work to it.
Just to feed back that @bepaald 's code found two messages in my backups which had attachments with bad MACs. When I looked at these messages in the Signal app they'd both failed to either send or receive. After deleting these messages in the app I created a new backup which I was able to restore from on a different phone.
If Signal could just handle restoring backups that include such messages...
@bepaald can you reupload the tool? I've got an issue where signal restores some, but not all messages.
@samizdis Good, I'm glad it worked. I assume the messages had attachments? It is usually the attachment that is bad, not the message itself. Did the tool happen to dump the attachment data? The fact that both messages had problems with them makes it seem it might not be a coincidence, maybe the backup process does not deal with failed messages properly. If you happen to know how the messages ended up in this failed state I might even be able to reproduce it.
@dcormier I agree, though if it is a bug in Signal, it would be even better if it didn't create faulty backups to begin with. If you (or anyone) has the problem that the original signal installation is no longer available (so that broken messages found by my code can't be deleted), I can probably make my program delete the problematic messages for you. I had this functionality in a previous version of my program and can probably hack it back in if needed.
@AJolly Naturally: https://send.firefox.com/download/d68364b5ad37f916/#FdeE7ZMaGIL97jZ7N7_98A
@bepaald I ran into the problem that I reinstalled my phone and I didnt expect the backup process to be a problem. I actually only need one picture from my backup. I had two backup files in Signal/Backups and both seem to fail at Frame 0. So i dont know if its this Bug or the files got corrupted. If its not too much work it would be very nice if you added the feature to delete problematic messages to your tool. Thank You :)
@griesenator Well a bad mac on frame 0 seems very strange to me. Are you absolutely 100% positive you have the passphrase correct? If you are, I'm afraid chances of restoring that backup may be very low. The first frame is not an attachment frame, it is not even a message yet, it is only the frame that holds the database version. It is a very small frame and it seems unlikely Signal will write it incorrectly. If corruption has occurred it probably will affect other frames as well making recovery very difficult. I have updated the link in the last post, if you run that program as follows: badmacinfo --dontbreak [backupfile] [passphrase], does it get any good frames after the first bad one? If so, I could probably get the program to just dump all your attachments to disk. I don't think dropping the bad frames is a good idea given the special nature of frame 0, but if that is the only bad frame, and we know the database version, I could probably also hack the program to write a new proper frame 0. Let me know what the output of the program is and we'll see if there is any use in trying to fix your database.
PS. Forgive me for not always answering quickly in this thread, I have busy periods now and then...
@bepaald Well what can I say it was my fault. Just found out i had two passphrases saved in my password manager. It really was just the wrong passphrase. Thank you!
Hey, I ran into a problem with my backup as well. I encrypts until 7501 messages, then cancels with "Wrong passphrase" message. If I change the passphrase the restoration is rejected immediately.
signal-back is able to restore the messages (and attachments). With this amount of data is was not able to check if all of them are restored, but some pretty recent ones are. If I use older backup files (that pre-date some of the attachments restored by signal-back) I run into the same error.
My old phone was a Samsung Galaxy S4 Mini running LineageOS 14.1 or 15.1.
My new phone is a Fairphone 2 with Fairphone Open 19.05.2 (Android 7.1)
I am able to make a backup on my new phone and restore it.
Let me know if I can help in any way to find the problem. So far I never was able to restore a signal backup when I needed it, and I would really like this feature to finally work reliably.
Would it help to try to import the backup on my old phone again?
Edit: I first thought there were only broken links for @bepaald 's tool, but now I found a working one. However, I don't feel confident to build it on my own... Is there any possibility to share a working binary for Linux or Windows?
@santa-klaus I've tried cross compiling a windows binary (again), but it doesn't seem to work. I get no errors during building, and it runs fine through wine, but when trying to run it on Windows 10 (in a VM) it just stops running before the programs natural end (without any errors).
Really, building is not that hard, I've added a little build script to this archive: https://send.firefox.com/download/00ff44f1a5ece664/#xFYQZ8JHGG0EyT7x6CBSKQ. For example, if you download and run the latest fedora live image (For example, this one), create a network connection and extract the files, you just have to run: sudo dnf install gcc-g++ cryptopp-devel sqlite-devel followed by sh BUILDSCRIPT.sh, and you are good to go. The same buildscript also works on my Arch machine (with deps installed), so it might work on other distro's as well.
If you manage to get it running, it would be interesting to see if it is the same message that has the bad mac on all your broken backups. Also, inspect the dumped file (I'm assuming it is some attachment) and compare it to the one in the signal install on your phone. And is there anything special about the message (as was the case for @samizdis' failed messages)?
@bepaald Thank you very much for the instructions. Thanks to your tips and the script I was able to build it (but only on the Fedora live system, getting all the dependencies for my linux mint was a pain in the ass, so I gave up)!
The program indeed dumps an attachment. It is different for different backups, and I can't identify anything special about those messages. I cannot delete them or compare them to the originals, since my original install is gone. Can your tool write a backup file without the corrupted messages? I tried the new file optional arguments but it never wrote anything but the dump of the corrupted attachment. Also, is it possible to know if this is the only corrupt message, or does your tool stop after the first corrupt?
Anyway, thank you very much! Your help is much appreciated.
@santa-klaus Try this: https://send.firefox.com/download/00ff44f1a5ece664/#xFYQZ8JHGG0EyT7x6CBSKQ.
It is hacked together in just 15 minutes and hardly tested, so it might need some more work, but I'll leave the testing to you ;-). The version you used would stop at the first bad frame (unless run with the hidden '--dontbreak' option), but this version will scan the entire file (don't add switches, they will be interpreted as input/output files in this version).
Too bad the original install is missing. Are the different backups incremental backups of the same installation (so newer ones contain at least the same messages as the older ones)? So some messages that are bad in one backup are ok in another? I think that would be some evidence against a bug in Signal and for some other source of data corruption on the phone.
Let me know if the program works. Others might be able to use it as well.
@bepaald Wow, thanks for the black magic! Didn't have to do any testing, just needed to overcome my noobiness and figure out how to call it properly. Now Signal imports the repaired backup file smoothly! The corrupted attachments were two relatively large files, both over 25 mb - a video and a pdf. Interesting enough completely different ones than the corrupted files in the earlier backup file, where a couple of pictures were affected. Interesting enough all those attachments should be present in both backup files, but seem only to be corrupted in either one. So either the signal backup-routine messed up at different points, or the data got corrupted somehow else.
Please let me know if you need some further diagnostics. How should we follow up to improve Signal's behaviour? A meaningful errormessage would be nice for a start (wrong passphrase is not really helpful), but maybe they could include the functionality to reject entries with bad MAC. Would that be a feature request?
Sony Xperia X -> Sony Xperia XA2 Plus.
Tried Twice. Both times says it has processed thousands of messages before stopping and saying the password is invalid. Guess me mum is starting off with a blank slate of messages. Should I really be trying to evangelise this app, I wonder...
@bepaald links are down again... a quick link refresh would be great. i ran an older version you had posted as a windows binary a month or 2 ago, got it to clean up some junk... but still wasn't working on my new phone. went through 39k messages and gave me the bogus wrong password thing. i've tried compiling an older link on ubuntu, just wasn't work out for me at all lol. could not figure out my dependency issues or where i needed flags. even tried building arch in a vmm to run it your way, bigger failure than the phone decryption. any assistance would be greatly appreciated.
@santa-klaus Happy to hear it worked. If the attachments are really very important to you, we could probably extract the uncorrupted ones from one backup and insert them into the other to get a full complete and working backup (while I say this I'm realizing how busy I've been the last few weeks... It is definitely possible though). I can't think of anything else to test on your devices, it really looks like some sort of memory corruption in your case, but I'm not sure we'll ever be certain. Are there storage integrity checkers for android? That might provide some confirmation.
A meaningful error message is certainly a good idea. I don't know if I stated this before, but the chances of the password being incorrect after even 1 frame is decoded successfully are slim to none. If the hash check suddenly fails after dozens (or hundreds or thousands) of messages it is by any reasonable measure impossible that the passphrase is wrong, so I would actually consider that toast a bug. It say 'corruption in backup file' or something to that effect. Also, rejecting frames with a bad mac (basically the functionality of my tool inside signal itself) would be a good feature request in my opinion (just remember feature requests go on the community forums, not on github's issue tracker). Though it is somewhat tricky to implement properly (ie better than I have done). In a way we've been lucky (though statistics are on our side) that all problems have been in attachment data, if corruption occurs in earlier frames (building the sql database or even before that), just dropping the frame might not give desired results.
@Ivoz Obviously this isn't meant to affect your stance towards signal as it nothing official, but just in case you haven't followed all of this thread: it might be possible to retrieve (most of) your mums messages using the tool linked below.
@oneof3holes Of course: https://send.firefox.com/download/00ff44f1a5ece664/#xFYQZ8JHGG0EyT7x6CBSKQ I think the easiest building instruction are probably in this comment from 8 days ago. Good luck!
@bepaald got fedora installed on virtualbox no problem. Installed dependencies and built via your script, super awesome of you to do that btw. However running the program i'm getting a segmentation fault while writing endframe that leads to a core dumping. The worst part is, your program absolutely identified the offending message, but I tried changing versions of signal on my old phone to try and work around the encryption issue long before i went online for answers. So I've lost access to my message and only have the backups I made. Can't re-register that phone since it's out of service and doesn't work on my new provider's bandwith lol, but i don't think the data itself was destroyed. I don't think i can root my old phone to dig deeper on it, could be wrong on that. I was able to get my texts and pics back with signal-back, but i really wish they were in the appropriate conversations.
Here's the message it ended on. If you have any thoughts or suggestions I'm willing to try it. I could just being shit out of luck here, so don't feel bad telling me that either. If I weren't so impatient and fiddling around on my old phone I could've solved it with your program by now.
Trying to dump decoded attachment to file 'attachment_4069.bin'
FRAME 38283 (089.7%)... done!
Removing 1 bad frames from database...
Exporting backup to 'newbackup.backup'
Dealing with table 'sms'...31522 entries...
Dealing with table 'mms'...2760 entries...
Dealing with table 'part'...1975 entries...
Dealing with table 'thread'...0 entries...
Dealing with table 'identities'...0 entries...
Dealing with table 'drafts'...0 entries...
Dealing with table 'push'...0 entries...
Dealing with table 'groups'...0 entries...
Dealing with table 'recipient_preferences'...0 entries...
Dealing with table 'group_receipts'...0 entries...
Dealing with table 'job_spec'...0 entries...
Dealing with table 'constraint_spec'...0 entries...
Dealing with table 'dependency_spec'...0 entries...
Dealing with table 'sticker'...0 entries...
Writing EndFrame...
Segmentation fault (core dumped)
@oneof3holes Whoops, well the segfault is certainly a bug. I'm afraid, looking at the output you pasted, I am guessing there is more corruption in the backup file than my tool can currently fix. The backup file is not read in its entirety (hence 0 'thread' entries, and segfaulting when trying to write the (unavailable) endframe). This probably means that not only frame data is corrupted, but also the byte in between the frames which holds the size of the next frame. Once this happens, no frame will be decoded properly anymore. To possibly confirm this, try this new version of the tool. It will not give better results, but might tell me why it stops decoding frames at 89.7% (and it hopefully will not segfault either).
I have a few possible fixes in mind already, for example, just guessing ('bruteforcing' in a way) the framesize, skipping it, and check if we get a valid frame after that (though more than just the next frame could be affected). But I will first need to try and recreate your situation on one of my own backups. Implementing will almost certainly take more than one sit down and I do not know when I will have time for even one programming session (though hopefully this weekend). How many backups do you have? Would you miss out on a lot if you use an older one, or do they have the same problem?
Trying to dump decoded attachment to file 'attachment_4069.bin'
FRAME 38283 (089.7%)... Failed to read next frame (951835245 bytes)
done!
Removing 1 bad frames from database...
Exporting backup to 'newbackup.backup'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing SqlStatementFrame(s)...
Dealing with table 'sms'...31522 entries...
Dealing with table 'mms'...2760 entries...
Dealing with table 'part'...1975 entries...
Dealing with table 'thread'...0 entries...
Dealing with table 'identities'...0 entries...
Dealing with table 'drafts'...0 entries...
Dealing with table 'push'...0 entries...
Dealing with table 'groups'...0 entries...
Dealing with table 'recipient_preferences'...0 entries...
Dealing with table 'group_receipts'...0 entries...
Dealing with table 'job_spec'...0 entries...
Dealing with table 'constraint_spec'...0 entries...
Dealing with table 'dependency_spec'...0 entries...
Dealing with table 'sticker'...0 entries...
Writing SharedPrefFrame(s)...
Writing EndFrame...
Error: EndFrame not found
@bepaald this was the error notification this go. I wish it weren't such sensitive data, otherwise I'd just send you my backup to play with. I am most certainly willing to try all the upgrades and tinkering you do to your program and post my results. This is my only backup I have, unfortunately. I had a little more faith in signal than I should have.
@oneof3holes Thanks. It is obvious the framesize is not correct here, it tries to read over 900mb of data for a single frame, which given the 100mb max for attachments should not be possible. I just realized it is also possible, though unlikely, that the previous frame size was already incorrect (causing both the bad Mac, and the next framesize to be read from a random file position). Can you get any info from the dumped attachment (it greatly depends on the filetype)? Does it seem unexpectedly large or small? Can you open it, and is there any corruption, and is that at the start or the end of the data?
I'll try to get started on a possible fix.
I had a little more faith in signal than I should have.
Just for the record, the location of the corrupted data in this case, outside of any encryption and not representing user data, just a single unencrypted byte written to memory, is another strong indication that no bug in Signal caused this. Still seems more likely corruption due to bad memory or something like that.
@bepaald > Thanks. It is obvious the framesize is not correct here, it tries to read over 900mb of data for a single frame, which given the 100mb max for attachments should not be possible. I just realized it is also possible, though unlikely, that the previous frame size was already incorrect (causing both the bad Mac, and the next framesize to be read from a random file position).
Total size of my backup file is only 436.9mb of data, so something is amiss. The .bin file is 350kb in size, looking my gallery the original picture was 3.75mb, but i'm sure it's getting compressed in signal.
Can you get any info from the dumped attachment (it greatly depends on the filetype)? Does it seem unexpectedly large or small? Can you open it, and is there any corruption, and is that at the start or the end of the data?
Being that it's a picture, I have no idea what I'd be looking for in hex editor since it's a bin. That area is really out of my field of knowledge... other than I do have a gui hex editor on my virtual feodra.
Just for the record, the location of the corrupted data in this case, outside of any encryption and not representing user data, just a single unencrypted byte written to memory, is another strong indication that no bug in Signal caused this. Still seems more likely corruption due to bad memory or something like that.
I'm almost 100% certain this was a picture message that got hung up and didn't send, even after multiple attempts, because I live in a highly rural area with spotty cell signal for data. That definitely doesn't go on signal. What all that has to do with things is unknown to me, but I see how it could cause errors, especially since it wasn't the first time I've had hung messages... or there is a reason i was replacing my phone. It was an s5, kinda ancient by cell phone standards. Almost 4 years for something bad to write itself. Been rocking it since textsecure days a couple phones ago.
@oneof3holes Try this (don't get your hopes up yet, it's highly experimental): https://send.firefox.com/download/d6451062b7827b11/#o2JXjrFrGmnxrnRmIFW0_g
Please let me know if it works!
Being that it's a picture, I have no idea what I'd be looking for in hex editor since it's a bin.
You could try to just rename the .bin to jpg and open it and see... sometimes that works (if it fails to open at all it is probably corrupted from the very beginning):
It is interesting that the corruption first occurs in a message that (initially) failed to send, @samizdis also had bad macs on failed messages. I do not see how it could affect the integrity of the following messages (or even the bytesize marker) though.
@bepaald no luck, back to ending with
Writing EndFrame...
Segmentation fault (core dumped)
I did try renaming that .bin to pretty much every picture related file extension I knew, but to no avail.
@oneof3holes Hm, that's unexpected. Could you give me a bit more of the output?
@bepaald here's the entire display, excluding key info. I xxxxxx'd out the message body and recipient number as well.
COUNTER: 3458556959
Reading backup file...
FRAME 38282 (089.7%)...
WARNING: Bad MAC in frame: theirMac: (hex:) 2c 39 83 f5 9c 91 bc fe a5 bc
ourMac: (hex:) 12 34 eb 14 44 e8 2c 33 71 74 b7 c6 a4 ed 1c 53 7b 68 b9 0b ec 38 34 a2 dd 02 db 45 e9 ab 70 03
WARNING: Bad MAC in frame, trying to print frame info:
Frame number: 38283
Type: ATTACHMENT
- row id : 1982 (8 bytes)
- attachment id : 1556336139925 (8 bytes)
- length : 358385 (8 bytes)
- attachment : (hex:) d3 3c 81 dc 51 91 a2 79 f4 0c e1 28 e8 24 91 a9 e7 76 98 61 b0 ef 2d 24 e7 ... (358385 bytes total)
Frame is attachment, it belongs to entry in the 'part' table of the database:
- _id : 1982
- mid : 4069
- seq : 0
- ct : image/jpeg
- name : (NULL)
- chset : (NULL)
- cd : (NULL)
- fn : (NULL)
- cid : (NULL)
- cl : (NULL)
- ctt_s : (NULL)
- ctt_t : (NULL)
- encrypted : (NULL)
- pending_push : 1
- _data : /data/user/0/org.thoughtcrime.securesms/app_parts/part-1063672323.mms
- data_size : 358385
- file_name : (NULL)
- thumbnail : /data/user/0/org.thoughtcrime.securesms/app_parts/part-158091836.mms
- aspect_ratio : 2
- unique_id : 1556336139925
- digest : (NULL)
- fast_preflight_id : 1019665070002254628
- voice_note : 0
- data_random : (hex:) e2 9a 01 92 3e 55 ce 77 68 78 bc e9 a4 fe 28 ee 9e 17 d5 18 d2 02 0c f9 c2 a6 54 aa 26 61 9e 63
- thumbnail_random : (hex:) 3b 8a 38 0e 18 e4 c3 91 9b fa 54 4d b1 52 c1 19 1f 25 f3 99 70 98 70 17 6d 5d 44 11 74 c9 85 80
- width : 1093
- height : 1944
- quote : 0
- caption : (NULL)
- sticker_pack_id : (NULL)
- sticker_pack_key : (NULL)
- sticker_id : -1
Which belongs to entry in 'mms' table:
- _id : 4069
- thread_id : 42
- date : 2019-04-26 23:35:39 -0400 (1556336139574)
- date_received : 2019-04-26 23:35:39 -0400 (1556336139632)
- msg_box : 88
- read : 1
- m_id : (NULL)
- sub : (NULL)
- sub_cs : (NULL)
- body : xxxxxxxxxx
- part_count : 1
- ct_t : (NULL)
- ct_l : (NULL)
- address : xxxxxxxxx
- address_device_id : (NULL)
- exp : (NULL)
- m_cls : (NULL)
- m_type : 128
- v : (NULL)
- m_size : (NULL)
- pri : (NULL)
- rr : (NULL)
- rpt_a : (NULL)
- resp_st : (NULL)
- st : (NULL)
- tr_id : (NULL)
- retr_st : (NULL)
- retr_txt : (NULL)
- retr_txt_cs : (NULL)
- read_status : (NULL)
- ct_cls : (NULL)
- resp_txt : (NULL)
- d_tm : (NULL)
- delivery_receipt_count : 0
- mismatched_identities : (NULL)
- network_failures : (NULL)
- d_rpt : (NULL)
- subscription_id : -1
- expires_in : 0
- expire_started : 0
- notified : 0
- read_receipt_count : 0
- quote_id : 0
- quote_author : (NULL)
- quote_body : (NULL)
- quote_attachment : -1
- shared_contacts : (NULL)
- quote_missing : 0
- unidentified : 0
- previews : (NULL)
Trying to dump decoded attachment to file 'attachment_4069.bin'
FRAME 38283 (089.7%)... done!
Removing 1 bad frames from database...
Exporting backup to 'newbackup.backup'
Dealing with table 'sms'...31522 entries...
Dealing with table 'mms'...2760 entries...
Dealing with table 'part'...1975 entries...
Dealing with table 'thread'...0 entries...
Dealing with table 'identities'...0 entries...
Dealing with table 'drafts'...0 entries...
Dealing with table 'push'...0 entries...
Dealing with table 'groups'...0 entries...
Dealing with table 'recipient_preferences'...0 entries...
Dealing with table 'group_receipts'...0 entries...
Dealing with table 'job_spec'...0 entries...
Dealing with table 'constraint_spec'...0 entries...
Dealing with table 'dependency_spec'...0 entries...
Dealing with table 'sticker'...0 entries...
Writing EndFrame...
Segmentation fault (core dumped)
@oneof3holes Thanks, I'll look into it. (For the future, just the output from the Trying to dump decoded attachment to file-line is fine.)
Just to be sure, you're not accidentally running the original, previous version are you? Because the output is completely the same (the segfault is back and even the FRAME 38283 (089.7%)... Failed to read next frame (951835245 bytes)-line is missing again) and I can't think why this would happen in the newer version.
Also maybe try this one, it should be the same as the last one, it just outputs a tiny bit more info (and fixes a typo): badmacinfo.zip
@bepaald definitely got a file to spit out that signal would at least fully decrypt. Only problem is none of the messages actually show up inside signal. I can even create a back up, it enumerates the correct number of messages, but it stil comes up empty if I delete everything and trying using the newer signal created backup as my restore point.
Trying to dump decoded attachment to file 'attachment_4069.bin'
FRAME 38283 (089.7%)... Failed to read next frame (951835245 bytes at filepos 411110199)
Starting bruteforcing offset to next valid frame...
Checking offset 107580 bytes
GOT GOOD MAC AT OFFSET 107584 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 0 frames... YEAH!
Got frame, breaking
FRAME 38284 (089.8%)... Failed to get valid frame from decoded data...
done!
Removing 1 bad frames from database...
Exporting backup to 'newbackup.backup'
Writing HeaderFrame...
Writing DatabaseVersionFrame...
Writing SqlStatementFrame(s)...
Dealing with table 'sms'...31522 entries...
Dealing with table 'mms'...2760 entries...
Dealing with table 'part'...1975 entries...
Dealing with table 'thread'...0 entries...
Dealing with table 'identities'...0 entries...
Dealing with table 'drafts'...0 entries...
Dealing with table 'push'...0 entries...
Dealing with table 'groups'...0 entries...
Dealing with table 'recipient_preferences'...0 entries...
Dealing with table 'group_receipts'...0 entries...
Dealing with table 'job_spec'...0 entries...
Dealing with table 'constraint_spec'...0 entries...
Dealing with table 'dependency_spec'...0 entries...
Dealing with table 'sticker'...0 entries...
Writing SharedPrefFrame(s)...
Writing EndFrame...
Done!
@oneof3holes Ok, that is too bad. NOTE, all of the following is speculation What happens is that we need to guess how many frames were skipped before we read a (possible) frame with a good MAC. But there is really no way of telling, except for just decoding the frame (the frame number is input for the decoder) and trying to see if it is a valid frame. It seems to me that the '0 zero frames skipped' scenario falsely passes the validation (after all, over a 100kb were skipped, that should be at least one frame). I am not at home right now, and do not have a computer at my disposal. I will try to make the validation more strict (if I can), but it won't be before tomorrow.
In the meantime, if you feel comfortable, you can try two things: SEE EDIT BELOW
In the file 'badmacinfo/filedecrytor/getframe.cc', right after the linestd::cout << "YEAH!" << std::endl;, add a line readingframe->printInfo(). Run it, nothing will have changed except some info on the falsely validated frame. It might help me make the validation better tomorrow. Delete any info you find too private (but I expect nothing, just garbage output).- In the same file, a few lines earlier change
uint skipped = 0;to start at 1 instead. This should at least skip the first false positive. Given that false positives are rare (I didn't encounter them in my testing), the next hit might be correct. note If the output is correct and complete(and worth importing into signal) the percentage will reach 100%, and at least the 'thread' table will have some entries (this is the table that holds the conversations that are shown on signals main screen, they are stored after all the messages and attachments in the database. That explains the behaviour you've seen after restoring the backup).
If I'm wrong about any of this (and I could be), or it gets too complicated to fix (especially without direct access to the backup), it might be possible to work with the part you are able to get now, we just need to fill the thread table with proper values, but many of those could be derived from the message tables. But... That's for later, if all else fails (keeps failing). Good luck!
EDIT:
I tried quickly changing the validation of the EndFrame-type (which I think is the problem), and also did step 1 already, so try this please: badmacinfo.zip. And please post the output. If it doesn't work, you could still try the second point I made above.
@bepaald got another full decrypt, but no actual restoration in my app. Tried method 2, got a segmentation dump after a lot of output, couldn't even scroll back to my original command. So here's what I have about where things went sideways with option 2.
Nothing wrong here, just the only output I see that is different than got startype, bit32 type, or got endtype.
GOT STARTTYPE
BIT32 TYPE
BIT32 TYPE
BIT32 TYPE
GOT STARTTYPE
GOT STARTTYPE
nope! :(
Checking if we skipped 2 frames... nope! :(
Checking if we skipped 3 frames... nope! :(
Checking if we skipped 4 frames... BIT32 TYPE
GOT ENDTYPE
Here's the 10 or so entries before dumping.
GOT STARTTYPE
GOT ENDTYPE
GOT ENDTYPE
GOT ENDTYPE
BIT32 TYPE
GOT ENDTYPE
GOT ENDTYPE
BIT32 TYPE
GOT ENDTYPE
BIT32 TYPE
Segmentation fault (core dumped)
@oneof3holes Sorry, this is getting frustrating.. Please try this new version, it will not fix anything, but I need some hint what's going on. I hope this will give some info (I've removed the STARTTYPE, BIT32 TYPE stuff as well, it never shows up in normal backups, because these types should not occur in them, but with the corruption of course any random data can appear), please post all output after Trying to dump decoded attachment.
Sorry this is progressing so slowly. I've created several backups and manually broke them in all the ways I guess yours may be broken and my program fixes all of them so far (except the one where I destroyed all the data from a certain point in the file, but at least the program then just exits normally and says it cant fix it...). I'm still hoping to make some progress, but I have to admit it is difficult.
EDIT I found tons of places where my code assumes correct input (it wasn't originally written to fix borken backups, so these are false assumptions in this case). It's going to take a long time tracking it all down and fixing them, so in the meantime, expect more segfaults... :)
Sorry this is progressing so slowly.
I'm beyond ecstatic you haven't grown tired of it, so no need to apologize to me. You're the one doing me a favor here.
Here's the output on the newest version
Trying to dump decoded attachment to file 'attachment_4069.bin'
FRAME 38283 (089.7%)... Failed to read next frame (951835245 bytes at filepos 411110199)
Starting bruteforcing offset to next valid frame...
Checking offset 107580 bytes
GOT GOOD MAC AT OFFSET 107584 BYTES!
Now let's try and find out how many frames we skipped to get here....
Checking if we skipped 0 frames... nope! :(
Checking if we skipped 1 frames... nope! :(
Checking if we skipped 2 frames... nope! :(
Checking if we skipped 3 frames... nope! :(
Checking if we skipped 4 frames... terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
I'm beyond ecstatic you haven't grown tired of it, so no need to apologize to me. You're the one doing me a favor here.
Well, trying anyway... No worries about growing tired of it, I'm quite determined to squeeze some useful bits out of your backup, just not sure how long it'll take.
So... the bad_alloc usually means running out of memory. I did find a small memory leak in the program, it shouldn't have been that bad after just 5 iterations of the loop.... but maybe things are worse because of the VM. Anyway, I hope I fixed that. Also fixed at least one segfault I was able to reproduce here by feeding the program random data. So try this one:
Again, output would be nice.
Same happens for me:
- Old mobile: Samsung S4 mini, most recent Lineage OS (unstable), Android 9
- New mobile: Sony XZ2 Compact Android 9
Unfortunatly, I was not able to build badmacinfo on Kubuntu 19.04 (with g++, libcrypto++-dev libcrypto++-doc libcrypto++-utils, libsqlite3-dev installed):
ryptbase/getbackupkey.cc:28:15: error: ‘digest’ was not declared in this scope
std::memcpy(digest, pass, passlength);
^~~~~~
cryptbase/getbackupkey.cc:28:15: note: suggested alternative: ‘_xtest’
std::memcpy(digest, pass, passlength);
@BenoMueller There surely was more to the error than you pasted... digest is defined in the line right above the line this error is generated from. The problem I believe is that Ubuntu for some reason is shipping with a very old version of cryptopp (5.6.4 I believe? Current version is 8.2.0), which does not define the CryptoPPnamespace. So I suggest you find some way to update it or run a different (I'm resisting the urge to say 'proper') distro, like Arch or Fedora (even just the live image would work, as I describe in this comment). A bit earlier in this thread, someone also had trouble with the sqlite version in Ubuntu, but I don't know the details about that.
I hope it works for you!
Again, output would be nice.
No output this time... it officially worked. I can't even begin to express my gratitude, but thank you.
The problem I believe is that Ubuntu for some reason is shipping with a very old version of cryptopp (5.6.4 I believe? Current version is 8.2.0), which does not define the CryptoPPnamespace. So I suggest you find some way to update it or run a different (I'm resisting the urge to say 'proper') distro, like Arch or Fedora (even just the live image would work, as I describe in this comment). A bit earlier in this thread, someone also had trouble with the sqlite version in Ubuntu, but I don't know the details about that.
Hysterically i was one of the people having that compilation issue. Running fedora in virtualbox was a relative ease. Ubuntu was running running a lower versiion of cryptopp and wouldn't not upgrade any further, despite throwing sudo and -dev and all sorts of things on my command. It honestly took less than 20 minutes to install fedora and have your program downloaded and compiling. I did have to use dropbox to transfer my files from phone to virtualbox as i never quite figured out sharing a directory or enabling a usb port lol.
Again, thank you so much!
Ok, with the 'proper' distro (I downloaded Fedora), it worked, thanks for the hint. I executed the following:
badmacinfo --dontbreak [backupfile] [passphrase]
Find attached the output:
Output_Badmacinfo.txt
Then I ran the following to generate the new backupfile without the broken attachment:
badmacinfo [backupfile] [passphrase] [newbackupfile]
And, YES, I could import the new backup file into Signal.
Many, many thanks for your help!
No output this time... it officially worked. I can't even begin to express my gratitude, but thank you.
No problem! I'm happy it finally worked, yours was certainly the most problematic backup so far, so it feels a bit like a victory, and I needed that. I hope you haven't lost too many messages.
@BenoMueller Excellent, happy to help! Thanks for reporting your success back here.
For anybody following this, or coming here in the future: Code will now be at: https://github.com/bepaald/signalbackup-tools
No problem! I'm happy it finally worked, yours was certainly the most problematic backup so far, so it feels a bit like a victory, and I needed that. I hope you haven't lost too many messages.
I lost the 1 offending picture message and i have a few weeks worth of picture messages that didn't transfer back into signal when i imported in from the default s10 messaging app, but other than we are good to go. And from my view point, huge win as well. I have maybe $40-50 usd in btc dust sitting in a wallet, I'd gladly send it your way. Probably crap pay for the work you did, but I'd feel like a jerk if I didn't offer you something for your time and effort.
@oneof3holes Good, I take it you haven't been using Signal while we were working on your backup then? Cause locally, I've succesfully merged threads from two different backups into one. We could have tried that to get newer messages into the old backup.
I've put a bitcoin link on bottom of the project page's readme. Feel free to donate anything you like, but know that includes nothing! I do this as a hobby, and did not anticipate any donations. I would have done it, and will continue to do it without any compensation.
@bepaald Exactly why I'd gladly give it. In case you're the nationality your name indicates, hit up a coffee shop for me or something. My chosen trade is construction, so I appreciate donating time for free to those who need it. But I only refuse money once from those I know can afford it.
Oh, I also emailed signal and informing them of your fix... and encouraging them to incorporate it lol. Nothing super important in my texts, but can't put a price on years of discussions. And I'll monitor this and your git if you upgrade and do fancier things, since my file was such a pain in the ass. Just tag me and I'll try to get back in under a day.
And honestly, may ditch ubuntu for fedora, really liking it. I aborted arch because a lot of the guis were fedora similar. Apparently I may have been wrong for not wanting to explore that further, but I've learned a lot thanks to this thread. For that and a fix to my backup, I'm eternally grateful.
@bepaald I will try to write up a feature request for Signal to reject bad frames in the backup in the forum. As for fixing the bug (wrong error message), do you think we should open a new issue stating explicitely that?
Also, I might want to try to merge different backups. My Signal lost most attachments when I moved it from SD-card to internal storage (my SD card is behaving funky), so I am left with a new (working) backup without most pictures, but about three weeks of new messages, and the old backup your tool fixed with working messages. Don't know when I'll find the time to try it, and it's not super important for me since I have those pictures still in my desktop app, but I think I would give it a try if you need a tester. Just let me know what I should do 😁
Thanks for sharing your tool on github, and thanks again for all the work!
@santa-klaus Yeah, I think a new issue would be in order. Just because this one is so old and long, I don't think any dev is still subscribed to this one so they might not read it here.
I'll get to work on the merging part as soon as I have some time. Much code is already written, I just need to merge it into the current latest version (hopefully that's not too hard). And I need to write a proper argument parsing routine, as the tool is getting too much functionality for the simple parsing it's doing now. I don't know when I'll have time, but I'll let you know as soon as I have something to test.
I found this issue because I also encountered an 'Invalid Passphrase' error message when trying to restore a Signal backup on my new phone that was created with my old one. I tried to use the tools https://github.com/xeals/signal-back and https://github.com/bepaald/signalbackup-tools without success (more on this later; my situation is a bit different to the ones described above in the sense that not single bad frames were found but the backup file was just unreadable by the tools).
However, once I deactivated signal-backups on the old phone and re-enabled it (settings - chats and media - chat backups; don't forget to write down the new passphrase), Signal on the new phone was suddenly able to restore the backup successfully. Also the aforementioned tools where able to read the backup.
Since this might be of help, here is a short documentation.
Old phone: Moto G 4G (peregrine, LineageOS version 14.1.20190207-NIGHTLY-peregrine, Signal 4.43.8)
New Phone: Moto G7 Plus (Android 9, Build number PPW29.98-66, Signal 4.43.8)
To run the two tools mentioned above, I installed Fedora Mate (Release 30 (MATE-Compiz) 64-bit) in a virtual box.
Here are the versions of several tools:
[user@localhost ~]$ gcc --version
gcc (GCC) 9.1.1 20190503 (Red Hat 9.1.1-1)
user@localhost ~]$ sqlite3 --version
3.26.0 2018-12-01 12:34:55 bf8c1b2b7a5960c282e543b9c293686dccff272512d08865f4600fb58238alt1
[user@localhost ~]$ yum info cryptopp | grep -C2 Version
Installed Packages
Name : cryptopp
Version : 8.2.0
Release : 1.fc30
Architecture : x86_64
--
Available Packages
Name : cryptopp
Version : 8.2.0
Release : 1.fc30
Architecture : i686
[user@localhost share-Fedora]$ ./signal-back_linux_amd64 --version
signal-back v0.1.7-alpha.2-dirty
proto commit: d6610f0
[user@localhost signalbackup-tools]$ git log --pretty=format:'%h' -n 1
53e58af
Output when I run signalbackup-tools on a backup that was created when I first enabled Signal-backups (I took a picture of the passphrase and check it several times):
[user@localhost signalbackup-tools]$ ./signalbackup-tools ../signal-2019-07-25-15-15-17.backup [passphrase]
IV: (hex:) 55 55 04 65 fa c7 4b 38 f2 3e ea ac 22 b3 9d 7c (size: 16)
SALT: (hex:) 4a 68 61 12 1b 19 75 98 a7 a2 10 74 e9 94 2e c4 9f a1 9d eb be ce ce aa 17 91 81 5b 59 34 d0 9e (size: 32)
BACKUPKEY: (hex:) 29 59 28 52 e0 d4 b1 a8 aa 35 d8 54 4a b8 46 64 c1 70 92 26 d0 29 68 22 65 05 c4 b8 0f ba 6d 81 (size: 32)
CIPHERKEY: (hex:) 72 32 f6 1b b8 ee 7b 5f 2d 1b db af e6 ed bd 8e 1d 89 66 47 b4 8e 0e 79 0c e3 38 a6 68 67 cc 7c (size: 32)
MACKEY: (hex:) bb f4 73 02 32 6a 16 d5 6d ce 65 89 17 c4 19 a8 6a 8c aa c4 76 2e fe 4f a9 cc 1c 5c 44 06 43 9a (size: 32)
COUNTER: 1431635045
Reading backup file...
FRAME 0 (000.0%)...
WARNING: Bad MAC in frame: theirMac: (hex:) c3 e4 32 15 47 d5 c6 88 7c 4e
ourMac: (hex:) d9 70 16 25 db 07 45 2d 9d 33 55 dc b7 07 10 e6 e5 c3 6c 97 2d 6a 9a 85 a9 a7 a5 99 bd 85 7d 73
Failed to get valid frame from decoded data...
Data: (hex:) 60 f5 71 01
done!
Output with signal-back:
user@localhost share-Fedora]$ ./signal-back_linux_amd64 check signal-2019-07-25-15-15-17.backup
Password: error: Encountered error while checking: failed to write raw: consume [attachment]: can't read attachment of length 0
Output of the tools when I once deactivated Signal backups on the old phone and re-enabled it:
[user@localhost signalbackup-tools]$ ./signalbackup-tools ../signal-2019-07-25-22-02-08.backup [passphrase]
IV: (hex:) 7c 0b 53 20 74 18 b3 cc 8e 58 d1 7d 7a 5a 4a 49 (size: 16)
SALT: (hex:) 2c 3d ae c5 2b d0 55 1d 71 aa a0 7c ca e6 eb 0b b3 23 a1 6f e1 a0 d2 f2 60 5e 7c a1 fd 02 d3 0c (size: 32)
BACKUPKEY: (hex:) 32 e5 ff d4 1d 0d 7e e6 79 89 0f 3a 8e 69 25 96 fd 8c 46 e3 8c fa ae 98 9c e0 da e6 46 68 ae 95 (size: 32)
CIPHERKEY: (hex:) 1c 0f 87 25 eb d4 f4 7e ba 94 d5 d5 55 c3 59 3c 68 d4 e6 05 eb c3 9b 73 50 04 fa be 61 8d c3 7e (size: 32)
MACKEY: (hex:) 39 e8 28 82 bd 57 fa 7e 5c 7c 1b 9a 17 58 a4 67 11 e0 d3 71 83 fb 5a b8 24 33 6c 6d 2c 3e ce 08 (size: 32)
COUNTER: 2081116960
Reading backup file...
FRAME 15660 (100.0%)... Read entire backup file...
done!
[user@localhost signalbackup-tools]$ cd ..
[user@localhost share-Fedora]$ ./signal-back_linux_amd64 check signal-2019-07-25-22-02-08.backup
Password: 2019/07/28 22:34:13 Backup looks okay from here.