stefankueng/CryptSync

Two Useful Features to improve speed: compression level and split size, as 7zip parameter

Gianbo opened this issue · 3 comments

I'd like to see two features that can greatly improve time of compression and transmission over cloud when large set of files are managed.
These are already implemented in 7zip, so there's only the need to add them as settings in the app (or defined in a editable configuration file):

  1. Compression level . Value 1 to 5 (instead of 9) will greatly improve processing speed when large set of files are managed (added/replaced).
  2. Split size of 7zip archive. E.g. 1-100Mb would be much more efficient for transmission to/from the cloud.

Setting these two parameters would greatly improve overall speed of operation for large set of files, which is a typica use case for me.

  1. you can already configure file types to not get compressed. And for file types that can be compressed the highest value is always preferred
  2. splitting is implemented in 7zip.exe but not in the compression lib itself. So it's not just "adding a setting". Also: why do you think splitting would be more efficient for transmission? If only one byte in a file changes, that would still change all the splitted file parts so they would still have to be transmitted.

Thank you for the reply Stefan.
I've asked these features because the syncronization to the cloud for me takes hours (folders with large set of files), and this could be greatly reduce actin on compression time and transfer of large files. I explain below the reasons.

For lzma an increase of 20-30% of compression on level 9 implies a time 6x larger than level1, which becomes a major drawback to sync large set of files in the cloud.
You can see typical graph for compression/time of lzma for instance here:
https://quixdb.github.io/squash-benchmark/unstable/

Transferring small files to the cloud is always more efficient than a large one because multiple connenctions are created only in the first case.
In fact, such solution is also adopted by a similar opensource solution, you probably know, cryptomator:
https://github.com/cryptomator/cryptomator
cryptomator encrypt anything as small files and has the ability to display the folder in the cloud as a mounted volume (as truecrypt/veracrypt without the harassing of a single large volume file) but imho the code is still very buggy...

Having at least the ability to set compression level to 1 (without splitting, if not supported by lzma lib) should allow to make cryptsync much faster at expense of negligible increase of cloud space...

2. splitting is implemented in 7zip.exe but not in the compression lib itself. So it's not just "adding a setting". Also: why do you think splitting would be more efficient for transmission? If only one byte in a file changes, that would still change all the splitted file parts so they would still have to be transmitted.

Stefan,
I also think that this feature would be very useful for two reasons:

  1. Indeed, many clouds support the transfer of several files at the same time. Thus, uploading and downloading a large file will speed up.
  2. This is probably the most important. Splitting a large file will allow us to bypass the file size limit of some cloud providers and older file systems.
    P.S. I have been using your program for many years, very useful and reliable.