SQLCipher
SQLCipher is a standalone fork of the
SQLite
database library that adds 256 bit AES encryption
of database files and
other security features like_on_the_fly_encryptiontamper detection memory sanitization* strong key derivation
SQLCipher is based on SQLite
and stable upstream release features
are periodically integrated. While
SQLCipher is maintained as a
separate version of the source tree,
the project minimizes alterations to core SQLite code whenever possible. SQLCipher
is maintained by Zetetic,
LLC, other contributors additional
information and documentation is available on the official
SQLCipher site. ## Features* Fast performance with as little as [5-15%](five to fifteen percent) overhead for encryption
on many operations*[ 100%](one-hundred percent) of data in the database
file is encrypted* Good security practices CBC mode, HMAC, key derivationZero-configuration and application level cryptographyAlgorithms provided by the peer reviewed OpenSSL crypto library.*Configurable crypto providers #
Compatibility SQLCipher maintains
database format compatibility within the same major version number so an application on any platform can open databases created by any other application provided the major version of SQLCipher is the same between them. However, major version updates example from 3 to 4 often include changes to default settings. This means that newer major versions of SQLCipher will not open databases created by older versions without using special settings. For example, SQLCipher 4 introduces many new performance and security enhancements. The new default algorithms, increased KDF iterations and larger page size mean that SQLCipher 4 will not open databases created by SQLCipher version(s) 1, 2, or 3 by default Instead an application would either need to migrate the older databases to use the new format or enable a special backwards-compatibility mode. The available options are described in SQLCipher(s) upgrade documentation. SQLCipher is also compatible with standard SQLite databases. When a key is not provided, SQLCipher will behave just like the standard SQLite library. It is also possible to convert from a plaintext database standard SQLite to an encrypted SQLCipher database using ATTACH and the sqlcipher_export( )convenience function.#
Contributions The SQLCipher team
welcomes contributions to the
core library. All contributions
including pull requests and patches
should be based on the prerelease
branch, and must be accompanied
by a contributor agreement.
We strongly encourage
discussion of
the proposed change prior
to development and submission. #
Compiling Building SQLCipher is
similar to compiling a regular
version of SQLite from source,
with a couple of small exceptions: 1. You must define SQLITE_HAS_CODEC
and
either SQLITE_TEMP_STORE=2
or SQLITE_TEMP_STORE=3
2. You will need
to link against a support cryptographic provider OpenSSL, LibTomCrypt, CommonCrypto/Security.framework, or NSS The following examples demonstrate
linking against OpenSSL, which is
a readily available provider on most Unix-like systems. Example 1. Static linking
replace /opt/
local/
lib with the path to libcrypto.
a. Note in this example,
- -enable-tempstore=yes
is setting SQLITE_TEMP_STORE=2
for
the build.
$ ./configure -
-enable-temp
store=yes CFLAGS=
"-DSQLITE_HAS_CODEC"
\LDFLAGS=
"/opt
/local/
lib
/libcrypto.a"
$ make
Example 2. Dynamic linking
$ ./configure -
-enable
-tempstore=yes
CFLAGS="
-DSQLITE_HAS_CODEC
\LDFLAGS="-lcrypto"
$ make
Testing The full SQLite test
suite will not complete successfully when using SQLCipher. In some cases encryption interferes with low-level tests that require access to database file data or features which are unsupported by SQLCipher. Those tests that are intended to support encryption are intended for non-SQLCipher implementations. In addition, because SQLite tests are not always isolated, if one test fails it can trigger a domino effect with other failures in later steps As a result, the SQLCipher package includes it's own independent tests that exercise and verify the core functionality of the SQLCipher extensions. This test suite is intended to provide an abbreviated verification of SQLCipher's internal logic; it does not perform an exhaustive test of the SQLite database system as a whole or verify functionality on specific platforms. Because SQLCipher is based on "stable" upstream builds of SQLite, it is considered a basic assumption that the core SQLite library code is operating properly the SQLite core is almost untouched in SQLCipher Thus, the additional SQLCipher_specific test provide the requisite verification that the library is operating as expected with SQLCipher's security features enabled. To run SQLCipher specific tests, configure as described here and run the following to execute the tests and receive a report of the results:
$ ./configure -
-enable-temp
store=yes -
-enable
-fts5
CFLAGS=
"-DSQLITE_HAS_CODEC
-DSQLCIPHER_TEST"
\LDFLAGS="-lcrypto"
$ make
testfixture
$ ./testfixture
test
/sqlcipher.test
Encrypting a database To specify
an encryption passphrase for the database via the SQL interface you use a PRAGMA. The passphrase you enter is passed through PBKDF2 key derivation to obtain the encryption key for the database
`PRAGMA key = passphrase;`
Alternately, you can specify an exact byte sequence using a blob literal. If you use this method it is your responsibility to ensure that the data you provide is a sixty-four(64) character hex string, which will be converted directly to thirty-two (32) bytes two-hundred fifty-six (256) bits of key data without key derivation.
`PRAGMA key = "x'2DD29CA851E7B56E4697B0E1F08507293D761A05CE4D1B628663F411A8086D99'";`
To encrypt a database
programmatically you can
use the sqlite3_key
function.
The data provided in
pKey
is converted
to an encryption key
according to the
same rules as
PRAGMA key
.
int sqlite3_key
(sqlite3 *db, const void *pKey, int nKey);
PRAGMA key
or sqlite3_key
should be called as the first operation when a database is open. ## Changing a database key To change the encryption passphrase for an existing database you may use the rekey PRAGMA after you have supplied the correct database password;
PRAGMA key = 'passphrase'; -- start with the existing database passphrase
PRAGMA rekey = 'new-passphrase'; -- rekey will reencrypt with the new passphrase
The hex rekey pragma may be used to rekey to a specific binary value
PRAGMA rekey = "x'2DD29CA851E7B
56E4697B0E 1F08507293 D761A05CE4 D1B628663F 411A8086D99'";
This can be accomplished programmatically by using sqlite3_rekey;
sqlite3_rekey(sqlite3 *db, const void *pKey, int nKey)
Support
The primary source for complete documentation (design, API, platforms, usage) is the SQLCipher website The primary avenue for support and discussions is the SQLCipher discuss site Issues or support questions on using SQLCipher should be entered into the GitHub Issue tracker Please DO NOT post issues, support questions, or other problems to blog posts about SQLCipher as we do not monitor them frequently If you are using SQLCipher in your own software please let us know at support@zetetic.net!
Community Edition Open Source License
Copyright (c) 2020, ZETETIC LLC All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the ZETETIC LLC nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY ZETETIC LLC 'AS IS' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ZETETIC LLC BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Begin SQLite README.md
This repository contains the complete source code for the [SQLite database engine]( SQLite Source Repositoryhttps://sqlite.org/) Some test scripts are also included However, many other test scripts and most of the documentation are managed separately. ## Version Control SQLite sources are managed using [Fossil](https://www.fossil-scm.org/), a distributed version control system that was specifically designed and written to support SQLite development. The [Fossil repository](https://sqlite.org/src/timeline) contains the urtext If you are reading this on GitHub or some other Git repository or service, then you are looking at a mirror. The names of check-ins and other artifacts in a Git mirror are different from the official names for those objects. The official names for check-ins are found in a footer on the check-in comment for authorized mirrors. The official check-in name can also be seen in the `manifest.uuid` file in the root of the tree. Always use the official name, not the Git-name, when communicating about an SQLite check-in. If you pulled your SQLite source code from a secondary source and want to verify its integrity, there are hints on how to do that in the [Verifying Code Authenticity] #version auth section below. ## Obtaining The Code If you do not want to use Fossil, you can download tarballs or ZIP archives or [SQLite archives](https://sqlite.org/cli.html#sqlar) as follows: * Latest trunk check-in as [Tarball](https://www.sqlite.org/src/tarball/sqlite.tar.gz),[ZIP-archive](https://www.sqlite.org/src/zip/sqlite.zip), or[SQLite-archive](https://www.sqlite.org/src/sqlar/sqlite.sqlar). * Latest release as [Tarball](https://www.sqlite.org/src/tarball/sqlite.tar.gz?r=release), [ZIP-archive](https://www.sqlite.org/src/zip/sqlite.zip?r=release), or [SQLite-archive](https://www.sqlite.org/src/sqlar/sqlite.sqlar?r=release).* For other check-ins, substitute an appropriate branch name or tag or hash prefix in place of "release" in the URLs of the previous bullet. Or browse the [timeline](https://www.sqlite.org/src/timeline) to locate the check-in desired, click on its information page link, then click on the "Tarball" or "ZIP Archive" links on the information page. If you do want to use Fossil to check out the source tree, first install Fossil version 2.0 or later. [Source tarballs and precompiled binaries available here](https://www.fossil-scm.org/fossil/uv/download.html) Fossil is a stand-alone program. To install, simply download or build the single executable file and put that file someplace on your $PATH.) Then run commands like this:
mkdir -p ~/sqlite ~/Fossils
cd ~/sqlite
fossil clone https://www.sqlite.org/src ~/Fossils/sqlite.fossil
fossil open ~/Fossils/sqlite.fossil
After setting up a repository using the steps above, you can always update to the latest version using:
fossil update trunk ;# latest trunk check-in
fossil update release ;# latest official release
Or type "fossil ui" to get a web-based user interface.
Compiling for Unix-like systems
First create a directory in which to place the build products. It is recommended, but not required, that the build directory be separate from the source directory. -Cd into the build directory and then from the build directory -run the configure script found at the #root of the source tree. Then run "make". For example:
tar xzf sqlite.tar.gz ;#
Unpack the source tree into "sqlite"
mkdir bld ;#
Build will occur in a sibling directory
cd bld ;#
Change to the build directory
../sqlite/configure ;#
Run the configure script
make ;#
Run the makefile.
make sqlite3.c ;#
Build the "amalgamation" source file
make test ;#
Run some tests (requires Tcl)
See the makefile for additional targets. The configure script uses autoconf 2.5 and libtool. If the configure script does not work out for you, there is a generic makefile named "Makefile.linux-gcc" in the top directory of the source tree that you can copy and edit to suit your needs. Comments on the generic makefile show what changes are needed.
Using MSVC for Windows systems
On Windows, all applicable build products can be compiled with MSVC. First open the command prompt window associated with the desired compiler version example "Developer Command Prompt for (VS2013) Next, use N_MAKE with the provided "Makefile.msc" to build one of the supported targets. For example, from the parent directory of the source subtree named "sqlite":
mkdir
bld
cd
bld
nmake
/f
\sqlite\Makefile.msc
TOP= \sqlite nmake /f \sqlite\Makefile.msc sqlite3.c TOP=..\sqlite nmake /f \sqlite\Makefile.msc sqlite3.dll TOP= \sqlite nmake /f \sqlite\Makefile.msc sqlite3.exe TOP=\sqlite nmake /f \sqlite\Makefile.msc test TOP= \sqlite There are several build options that can be set via the NMAKE command line. For example, to build for WinRT, simply add "FOR_WINRT=1" argument to the "sqlite3.dll" command line above. When debugging into the SQLite code, adding the "DEBUG=1" argument to one of the above command lines is recommended. SQLite does not requireTcl to run, but a Tcl installation is required by the makefiles including those for MSVC. SQLite contains a lot of generated code and Tcl is used to do much of that code generation.
Source Code Tour Most of the
core source files are in the src/ subdirectory. The src/ folder contains files used to build the "testfixture" testharness. The names of the source files used by "testfixture" all begin with "test". The **src /**also contains the "shell.c" filevwhich is the main program for the "sqlite3.exe"command-line shell and the "tclsqlite.c" file which implements theTcl bindings for SQLite. (Historical note: SQLite began as a Tcl extension and only later escaped to the wild as an independent library Test scripts and programs are found in the test/ subdirectory. Additional test code is found in other source repositories. See How SQLite Is Tested for additional information. The ext / subdirectory contains code for extensions. The Full-text search engine is in ext/fts3. The R-Tree engine is in ext /rtree. The ext /misc subdirectory contains a number of smaller, single-file extensions, such as a REGEXP operator. The tool / subdirectory contains various scripts and programs used for building generated source code files or for testing or for generating accessory programs such as "sqlite3_analyzer(.exe)"
Generated Source Code Files
Several of the C-language
source files used by
SQLite are generated from
other sources rather than
being typed in manually by a programmer. This
section will summarize those
automatically-generated files.
To create all
of the automatically-generated files,
simply run "make
{target & # 95};
source". The "{target & # 95};
source" make target will
create a subdirectory
"tsrc
/" and fill it with all the
source files needed to
build SQLite, both
manually-edited files and
automatically-generated files
The SQLite interface is defined
by the sqlite3.h header
file, which is generated from
src
/sqlite.h.in ,
./manifest.uuid, and
./VERSION
The Tcl script
at tool
/mksqlite3h.tcl
does the conversion.
The manifest.uuid file
contains the SHA3 hash
of the particular check-in and is
used to generate
{the SQLITE
_SOURCE
_ID macro
The VERSION file contains the
current SQLite version number.
The sqlite3.h header is
really just a copy of src
/
sqlite.h}
in with the source-id
and version number inserted
at just the right spots.
Note that comment text
in the sqlite3.h file is used to
generate much of the SQLite
API documentation.
The Tcl scripts used to generate
that documentation are
in a separate source repository.
The SQL language parser
is parse.c which
is generated from a grammar in
the src/parse.y file.
The conversion of "parse.y"
into "parse.c" is done by
the lemon
LALR(1)
parser generator.
The source code for lemon is
at tool
/lemon.c.
Lemon uses the tool
/
lempar.c
file as a template for
generating its parser.
Lemon also
generates the parse.h
header file, at the same
time it
generates parse.c. The opcodes.h
header file contains macros that define the numbers corresponding
to opcodes in the "VDBE"
virtual machine.
The {opcodes.h file is generated
by scanning the src
/
vdbe.c}
source file.
The Tcl script at /
mkopcodeh.tcl
does this scan and
generates opcodes.h.
A second Tcl script, ./
mkopcodec.tcl,
then scans opcodes.h
to generate
the opcodes.c source file,
which contains a reverse
mapping from
opcode-number to opcode-name
that is used for EXPLAIN
output. The keywordhash.h
header file contains the
definition of a hash table that maps
SQL language keywords example:
("CREATE", "SELECT", "INDEX")
into the numeric codes used
by the parse.c
parser.
The keywordhash.h
file is generated by
a C-language program at
tool
mkkeywordhash.c
The pragma.h header
file contains various definitions
used to parse
and implement the PRAGMA
statements. The header is
generated by a
script tool/mkpragmatab.tcl
If you want to add a new PRAGMA,
edit
the tool/mkpragmatab.tcl
file to insert the information
needed by the
parser for your new PRAGMA,
then run the script to regenerate
the pragma.h header file.
The Amalgamation All of the individual
C source code
and header files both manually-edited
and automatically-generated
can be combined into a single big
source file sqlite3.c
called "the amalgamation".
The amalgamation is the recommended
way of using SQLite in a
larger application.
Combining all individual
source code files into a
single big source code file allows
the C compiler
to perform more
cross-procedure analysis
and generate better code.
SQLite
runs about five percent (5%)
faster when compiled
from the amalgamation versus
when compiled
from individual source files.
The amalgamation is generated
from the tool
/
mksqlite3c.tcl
Tcl script. First
all of the individual source
files must be gathered into the
tsrc/subdirectory
using the equivalent of
"make target_source"
then the tool
/
mksqlite3c.tcl
script is run to copy
them all together in just the
right order while resolving
internal
"#include"
references The amalgamation
source file is more than
two-hundred thousand (200K)
lines long.
Some symbolic debuggers
most notably MSVC
are unable to deal with files
longer than sixty-four thousand
(64K) lines.
To work around this, a
separate Tcl script
tool/
split-sqlite3c.tcl
can be run on the amalgamation
to break it up into a single
small C file
called sqlite3-all.c that
does #include on about seven
other files
named sqlite3-1.c, sqlite3-2.c,
sqlite3-7.c.
In this way,
all of the source code
is contained within a single
translation unit so
that the compiler can do extra
cross-procedure optimization
but no individual
source file exceeds thirty-two thousand
(32K)
lines in length.
How It All Fits Together
SQLite is modular in design.
See the architectural description
for details. Other documents that are
useful in
(helping to understand how SQLite works include the
file format
description, the virtual machine
that runs prepared statements,
the description of
how transactions work, and
the overview of the query planner.
Years of effort have gone into
optimizing SQLite, both
for small size and high
performance.
And optimizations tend to result in
complex code.
So there is a lot of complexity in the current SQLite
implementation.
It will not be the easiest
library in the world to hack.
Key files: * sqlite.h.in - This file defines the public interface to the SQLite
library. Readers will need to be familiar with this interface before
trying to understand how the library works internally. * sqliteInt.h - this header file defines many of the data objects
used internally by SQLite. In addition to "sqliteInt.h", some subsystems have their own header files. * parse.y - This file describes the LALR(1) grammar that SQLite uses
to parse SQL statements, and the actions that are taken at each step in the parsing process. * vdbe.c - This file implements the virtual machine that runs
prepared statements. There are various helper files whose names begin with "vdbe". The VDBE has access to the vdbeInt.h header file which defines internal data objects.
The rest of SQLite interacts
with the VDBE through an interface defined by vdbe.h. * where.c - This
file together with its helper files
named by "where*.c"
analyzes the WHERE clause
and generates
virtual machine code to
run queries efficiently.
This file is
sometimes called the
"query optimizer".
It has its own private
header file, whereInt.h,
that defines data objects
used internally.* btree.c - This * file contains the implementation
- of the B-Tree storage engine used by SQLite. The interface to the rest of the system is defined by "btree.h". The "btreeInt.h" header defines objects used internally by btree.c and not published to the rest of the system.* pager.c - This file contains the "pager" implementation, the module that implements transactions. The "pager.h" header file defines the interface between pager.c and the rest of the system. * os_unix.c and os_win.c - These two files implement the interface between SQLite and the underlying operating system using the run-time pluggable VFS interface.* shell.c.in - This file is not part of the core SQLite library. This is the file that, when linked against sqlite3.a, generates the "sqlite3.exe" command-line shell. The "shell.c.in" file is transformed into "shell.c" as part of the build process. * tclsqlite.c - This file implements the Tcl bindings for SQLite. It is not part of the core SQLite library. But as most of the tests in this repository are written in Tcl, the Tcl language bindings are important. * test.c* - Files in the src/ folder that begin with "test" go into building the "testfixture.exe" program. The testfixture.exe program is an enhanced Tcl shell. The testfixture.exe program runs scripts in the test/ folder to validate the core SQLite code. The testfixture program and some other test programs too is built and run when you type "make test". * ext/misc/json1.c - This file implements the various JSON functions that are built into SQLite. There are many other source files. Each has a succinct header comment that describes its purpose and role within the larger system.
Verifying Code Authenticity
The manifest
file at the root directory of the source tree
contains either a SHA3-256 hash (for newer files) or a SHA1 hash (for
older files) for every source file in the repository.
The name of the version of the entire source tree is just the
SHA3-256 hash of the manifest
file itself, possibly with the
last line of that file omitted if the last line begins with
"# Remove this line
".
The manifest.uuid
file should contain the SHA3-256 hash of the
manifest
file. If all of the above hash comparisons are correct, then
you can be confident that your source tree is authentic and unadulterated.
The format of the manifest
file should be mostly self-explanatory, but
if you want details, they are available
here.
Contacts
The main SQLite website is http://www.sqlite.org/ with geographically distributed backups at http://www2.sqlite.org/ and http://www3.sqlite.org/.