Crash when rekey from a plain big db to a cypher one
xujw89 opened this issue · 15 comments
crash with v4.0.4 and same with the latest tag version.
The plain db is over 200MB, and one table in this db has over 2w+ rows. When call rekey(), the program crash.
The crash stack is like:
The failed line:
The data pointed by pzErrMsg is 0xcccccccc, and then crash
The option change may like this in codecext.c line 362:
wish helpful，thank you very much
I checked the code of sqlite3RunVacuumForRekey. It is indeed necessary to initialize the first parameter passed to it to NULL (as you did according to your code lines marked in yellow), because SQLite tries to free memory if it is not NULL - and that can possibly result in a crash. Of course, I will fix this. Thanks for reporting.
However, I don't know whether this solves the issue with encrypting your large database. Return code 11 means database file corrupted. No idea, why this happens. Have you done an integrity check on the plain database? Maybe the plain database already has a problem. If not, I would need a way to reproduce the issue somehow to be able to further analyze the issue.
The plain db is fine, after I checked it with "PRAGMA quick_check;". I donnot know why it returns 11 either. But because this db is just cache file, with the failed return code, I will rebuild a new one. This is my solution now. May be later, I will check why SQLite return database file corrupted.
If you want to reproduce this issue, I think this will happens with a large table(may over 20k+ rows). You may try it. If the failed code doesn‘t show up, I will try to analyze the issue further. If you are insterested, I may send this cach file to you with permission of my clients.
Thanks for you reply.
For the ChaCha20 encryption scheme the functions EncryptPageChaCha20Cipher and DecryptPageChaCha20Cipher in source file codec.c can possibly return the error SQLITE_CORRUPT (corresponding functions for other encryption schemes).
It would be good to know whether the error is produced by one of those functions or by some other SQLite code. If one of those functions is responsible for producing this error, then it would be good to know the circumstances (values of variables legacy, nReserved, reserved and so on). If you are able to debug your application, providing this information could help to identify the cause of the problem.
Of course, I'm interested in reproducing the problem and identifying the cause of it. Therefore I will try to reproduce the problem. However, it would certainly be easier to experiment with a database file for which it is already known that it shows the problem. That is, if you could provide a sample database file, that would be great.
I have had no problems rekeying ~900MB wxSQLite3 databases (using default encryption settings).
@resilar: Thanks for looking into this issue, too.
Just a few minutes ago I finished a similar test with a 1.5 GB database containing 25.000.000 rows in a table. Rekeying worked without problems. Except for the problem with uninitialized pointer variables (which I will fix ASAP) for error message strings causing a crash at least in debug mode in case an error (like out of memory) occurs in sqlite3RunVacuumForRekey, I wasn't able to reproduce the database corrupt error.
@xujw89: please provide a sample database showing the issue on your system for further inspection, if possible. Thanks.
Sorry for the late reply.
The SQLITE_CORRUPT stack is like：
And the SQL line is: INSERT INTO vacuum_db.'dispatch_history' SELECT*FROM"main".'dispatch_history';(dispatch_history is the large table)
The db file contains some user's data, and I am trying to apply for the permission. I will send the file to your email(github@telle-online.de) as soos as possible.
The function sqlite3RunVacuumForRekey is an almost verbatim copy of SQLite's internal implementation of the VACUUM SQL command with minor adjustments to allow to use this function to encrypt or decrypt a database. All content from a database is copied to another temporary database, and finally copied back.
From the callstack it is unfortunately impossible to deduce what goes wrong. One would have to intercept the calls to the underlying database page encryption/decryption functions to find out the circumstances under which they fail.
Regarding a sample database exposing the problem: maybe you could try to remove specific user data or overwrite them with random data. As long as the resulting database still shows the corruption problem on rekeying, that would be ok.
I can assure you that the sample database will be used only for identifying the cause of the problem, and will be deleted as soon as the problem is solved.
If you get permission to send the sample database, sending it via e-mail might not work depending on the size of the database. In that case please let me know your e-mail address and I will send instructions how you can upload the file to my server, if you don't have a server yourself onto which you can place the file for download.
Just for curiosity, does plain VACUUM; also trigger the crash?
The call stack shows that the 1734th page has a corrupt flag byte (199 while the valid range is 0..15). So at least it looks like the first byte of page 1734 containing page flags is indeed corrupted. I find this surprising because, at the same time, Poly1305 MAC of the page must be valid (otherwise the code would have not reached the flag decoding logic).
@resilar Great that you spotted the invalid flag byte value. The question is whether this invalid flag byte is detected on reading a page from the plain database (for which there is no Poly1305 MAC) or whether this happened under other circumstances (in which case it might indicate that the encrypt function failed for some unknown reason). However, in the former case I would have expected that the command PRAGMA quick_check; would have revealed the corrupted flag byte, but according to @xujw89 this was not the case.
Performing a VACUUM command on the plain database is a really good idea. If the plain database has a corrupted page, most likely VACUUM should fail with a database corrupted message.
@utelle I already got the permission for the file, and had sent the file to you email(github@telle-online.de). Thanks a lot for your work.
Thanks for providing the database file. I'll inspect it and will report back here about my findings.
I made a few quick checks with the database file provided by you to verify whether I can reproduce your problem. I tested with the SQLite shell binaries which are distributed with the releases of wxSQLite3. I tested with the 32bit and 64bit shell versions (without ICU support) of wxSQLite3 4.0.4 and 4.4.3 under Windows 7 Prof 64bit. I used the default encryption scheme, i.e. ChaCha20.
- Executing
PRAGMA quick_check... worked (Result: ok) - Executing
VACUUM... worked (no crash, no error message). - Executing
PRAGMA rekey='password';... worked (no crash, no error message; reopening encrypted database worked, too).
That is, unfortunately I wasn't able to reproduce your problem. If rekeying the database file fails for you reproducibly, there have to be differences - either in the build process of your application or in your system environment. Did you apply any changes to the source code of the wxSQLite3 encryption extension or SQLite itself? Are you using the default encryption scheme (ChaCha20) or another one? Which compiler are you using? For which platform are you building? Please provide details about your environment and your build process.
My project is compiled with vs2017 for a win32 application. I included the source code(sqlite3secure.c ) of wxsqlite3 into the .vcxproj.
Did you apply any changes to the source code of the wxSQLite3 encryption extension or SQLite itself?
--NO, I didn't apply any change.
Are you using the default encryption scheme (ChaCha20) or another one?
--I didn‘t change any default setting of wxsqlite3.
Which compiler are you using?
--vs2017. Toolset: Visual Studio 2017 - Windows XP (v141_xp)
For which platform are you building?
--windows
Thanks for providing information about your build environment.
The Windows binaries provided with wxSQLite3 releases were created with VS2015. I repeated my tests with VS2017 ... this worked for me, too. That leaves a single difference - the toolset: I used v141 while you used v141_xp.
I had to install the Windows XP related toolset, because I personally dropped XP support long ago. Do you really need XP support?
Anyway, I repeated my test with toolset v141_xp ... and that also worked for me. So, I'm at a loss. I don't have the slightest idea, what else might be different in your environment.
Did you try to rekey your database file with the SQLite3 shell? If not, please do. Please report back whether you reproducibly experience a crash using the SQLite3 shell tool, or only when using your own application.
Since I found no evidence that there is a flaw in the SQLite encryption extension coming with wxSQLite3, and since I didn't get any further feedback within almost 2 months, I'm going to close this issue.
Feel free to reopen the issue, if additional information can be provided.