Feature Request: Reverse Mode
usr42 opened this issue · 32 comments
Add an option similar to the --reverse option for encfs:
Normally EncFS provides a plaintext view of data on demand. Normally it stores enciphered data and displays plaintext data. With --reverse it takes as source plaintext data and produces enciphered data on-demand. This can be useful for creating remote encrypted backups, where you do not wish to keep the local files unencrypted.
For example, the following would create an encrypted view in /tmp/crypt-view.
encfs --reverse /home/me /tmp/crypt-view
You could then copy the /tmp/crypt-view directory in order to have a copy of the encrypted data. You must also keep a copy of the file /home/me/.encfs5 which contains the filesystem information. Together, the two can be used to reproduce the unencrypted data:
ENCFS5_CONFIG=/home/me/.encfs5 encfs /tmp/crypt-view /tmp/plain-view
Now /tmp/plain-view contains the same data as /home/me
Note that --reverse mode only works with limited configuration options, so many settings may be disabled when used.
(Quote from encfs man page)
Reverse mode really is a nice feature. However, with reverse mode, there is no permanent store for the ciphertext (everything is encrypted on the fly and not stored). That means that gocryptfs could not store the file and block headers permanently.
encfs disables both in reverse mode, but this comes at a security cost, and is impossible for gocryptfs because GCM encryption absolutely needs a block header.
Now gocryptfs could just generate new block headers for every access (they are random anyway). But this would mean that you get different ciphertext every time you read a file (it still decrypts to the same plain-text, of course). This would make it pretty painful for backup purposes.
In short, I don't see a way to do this in a useful way for gocryptfs.
GCM-SIV ( https://tools.ietf.org/html/draft-irtf-cfrg-gcmsiv-01 ) looks like a way this could be implemented. Reopening.
Added $100 bounty:
https://www.bountysource.com/issues/31871717-feature-request-reverse-mode
@fulldecent Question: Do you need a writeable reverse mount or is read-only good enough?
I am happy with read only. My use case is secure backups with rsync.
If somebody needs read-write, maybe that can be a separate issue.
Added another $20 to the bounty.
Just FYI: I have started working on this in the "reverse" branch. I'll travel abroad for two weeks now but I expect to have things in a testable state by the end of September.
Very cool, thanks for the update
This is essentially done. Fetch the latest "reverse" branch if you want to test it.
However, it uses the GCM-SIV Go reference implementation from github.com/agl/gcmsiv which is painfully slow at about 2 MB/s.
I am thinking about switching to AES-SIV, which has a reasonably fast Go implementation and the format is finalized (RFC 5297). Votes against?
This is essentially done. Fetch the latest "reverse" branch if you want to test it.
Awesome! I'm skimming through the code. When trying to run go get in the branch, I'm getting:
package github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse: cannot find package "github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse" in any of:
/usr/lib/go-1.6/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse (from $GOROOT)
/home/bgw/.golang/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse (from $GOPATH)
However, I don't know if this is because I'm doing something wrong. I'm not a Go developer.
I am thinking about switching to AES-SIV, which has a reasonably fast Go implementation and the format is finalized (RFC 5297). Votes against?
The code looks better maintained too, so it seems like a good decision. However, I don't know enough about the pros/cons of these different cryptographic methods to intelligently comment.
That error message does not make sense to me, but I just tried from an empty $GOPATH, this should work (note the second "go get ."):
go get github.com/rfjakob/gocryptfs
cd $GOPATH/src/github.com/rfjakob/gocryptfs
git checkout reverse
go get . # Pull the remaining dependencies
BTW, quickstart for how to use this:
gocryptfs -reverse -init DIR
gocryptfs -reverse DIR MNT
and if you want you can mount this again in forward mode and you'll get the plaintext files:
gocryptfs MNT MNT2
@rfjakob, Thanks that works! The procedure I was trying was:
- Clone the repository (manually, using
git) - Check out the reverse branch
- Run
go getinside that directory
Toying around with the cli, was able to encrypt and decrypt a file in a directory. Very cool!
One minor bit of feedback: When I first saw the cli output upon setup, it wasn't immediately clear to me that the master key was deterministic, though it seems that it is.
@bgw Does this mean that if you delete the folder and recreate it you get the same master key?
I have switched to AES-SIV and have merged everything to master. Speed is at > 60MB/s on my CPU without AES-NI, so that should be pretty usable.
I have tagged it as "v1.1-beta1" because there may still be small changes to the IV derivation, so I'd wait for "v1.1" final before archiving data.
No way: https://github.com/rfjakob/gocryptfs/blob/master/internal/configfile/config_file.go#L52
Whoops, I totally missed that the -reverse -init step creates a .gocryptfs.reverse.conf file. If I delete that file and reinitialize, the master key changes.
@rfjakob, If the master key is stored in this file, why do I need to enter the password every time I mount? (Ignore that, the master key is derived from the combination of this file and the password)
Does this mean that if you delete the folder and recreate it you get the same master key?
If you mount the same directory multiple times, the resulting encrypted mount is encrypted with the same master key every time. This should mean you can incrementally rsync that encrypted mount somewhere else.
Looks good, thanks for all the effort.
I am okay to mark this done for bounty purposes. Since some other people have money in, I'll wait for them to give the go ahead too.
I'm fine with marking the bounty as done.
@rfjakob Thanks for your great work. Please fill in the form at https://www.bountysource.com/issues/31871717-feature-request-reverse-mode so we may award bounty.
I'm quite happy with the reverse implementation now and I don't think there will be any more changes. The feature is released as v1.1-rc1 and I'm going to close the ticket.
Just a word to @fulldecent and @bgw: Having a bounty on a feature is not only great because you get the money. It shows you that people are genuinely interested in advancing the project, and the amount of the donation doesn't even have to be high for that. Having the bounty on this was very motivating, thank you guys for that, you are awesome!
@rfjakob Thank you! This is my first time working with a bounty. Their website seems to be down right now, and I'm not sure how to close it out. I imagine you will need to mark it complete and then we confirm or something like that. Please let me know when you have marked it complete from their perspective so I can figure out my next step there.
Do you need a writeable reverse mount or is read-only good enough?
@rfjakob, did you finally made the reverse mode read-only, or also writable ?
Thx 👍
It's read only
What would be needed to make it writable ?
It's missing implementations for Create, Truncate, Unlink, Rmdir etc. Most work is probably the file Write function (forward mode for reference):
gocryptfs/internal/fusefrontend/file.go
Line 348 in 996d2f1
Just curious, do you plan it ?
No as I don't have a use case for it
@rfjakob Well there is a valid use case for it. 2-way syncing from multiple locations to remote untrusted locations, without double the disk space cost, using reverse mode.
And #108 (adding flag to disable all randomness in both forward and reverse for AES-SIV) would allow this use case easily and also alleviate the huge terror those of us with filesystems with hundreds of thousands of files in huge directory trees have, with possible corruption/loss of .diriv files at the remote untrusted location effectively destroying our backups. All birds with one stone.
Read only --reverse option only good for local backups. But --reverwrite option would allow you not only to create local backups but also backups using cloud like (dropbox, google drive or yandex disk).
Here are few benefits having files unencrypted on your hard-drive. One of them is possibility to compress files by filesystem. Properly encrypted files does not support compression (obviously because of entropy == 1) and that is a limitation of default encryption mode of gocryptfs (or encfs for example). You can't compress encrypted folders!
If you have encrypted a lot of emails (maildir format) you can save up to 10 times of space for text files (emails or logs) if they compressed using filesystem. But you have to compress them before encryption (or using --reverse option)
The only problem with --reversewrite is not know what to do with conflict files. Conflict files which have been created by sync tool with broken names like "2xX3403-1fc (2)". In reversewrite mode you can't create corresponding unencypted file since encypted file name corrupted by conflict suffix "(2)". encfs prevents to create such files. But it may be not the optimal solution.
EDIT: I endup patching encfs properly handling conflicting files, and it works great!
Another use case of the writeable reverse mode: I've tried to use it to backup to Megasync, but Mega creates .megaignore files to track what to include/exclude within a folder, and with a read-only folder you can't create those files, so you can't start a backup of that folder