use a binary format for wallet files
PiRK opened this issue · 2 comments
The Electrum wallet files contain the entire transaction history for a wallet, and thus can reach significant sizes (tens of megabytes). The transaction They are currently saved as JSON text, sometimes encrypted.
This could all be improved significantly by using a more compact data format than JSON for disk storage. A structured binary file would reduce the disk space usage, and using a file format that allows for an easy access to some data without having to read/parse all of it would improve performance significantly.
The following formats come to mind:
- sqlite (database format)
- HDF5 (easy to use data format with built-in indexing, compression....); drawback: dependency on numpy (+300MB packages)
I did some profiling by tweaking the JSON output format. With a very large wallet (>100 MB), there are some potential gains by writing compact unsorted JSON:
diff --git a/electroncash/storage.py b/electroncash/storage.py
index fd872d1d1..08e685c00 100644
--- a/electroncash/storage.py
+++ b/electroncash/storage.py
@@ -120,8 +120,10 @@ class JsonDB(PrintError):
return
s = json.dumps(
self.data,
- indent=4 if self.output_pretty_json else None,
- sort_keys=self.output_pretty_json,
+ indent=None,
+ sort_keys=False,
cls=util.MyEncoder
)
s = self.encrypt_before_writing(s)
sort,indent
| 15.489| |03| [profiler] WalletStorage.write 0.9943
no sort, indent
| 16.008| |03| [profiler] WalletStorage.write 0.8581
no sort, no indent
| 15.378| |05| [profiler] WalletStorage.write 0.4795
Related discussion: spesmilo#4823