Compress file content with `zstd`
robherley opened this issue · 3 comments
Hi Rob,
I was thinking of encoding the contents in the Upload() in internal/ssh/handler.go, and decoding the contents in DownloadFile() in internal/ssh/handler.go and FileHandler() in internal/http/handler.go.
I had two questions before working on the task:
- We now have two files sizes, the file size for the compressed file and the file size for the uncompressed file. Which should we consider when limiting the file size when uploading?
- Should we use stream encoding or block encoding? I don't have knowledge on which is better, but the doc says block encodes should be used for small encodes.
Instead of doing the encoding/decoding in the handlers directly, I was thinking about making the file's Content private, then having setters/getters to control the inflate/deflate. We access the content in a few places (tui, frontend renderer) so having something on the snips.File itself probably makes the most sense.
We can use the default compression ratio (3) and I think block encoding is perfectly fine for now, since we read the entire snippet into memory before writing.
As for the upload limit, I think we can still go with the uncompressed size, and we'll persist that uncompressed size in the database too. Less confusing for the end user.
The important thing here is we want to prevent breaking any other snips installations and we can dynamically inflate depending on some the file's content. Like mentioned before, a getter like GetContent() method can peek at the file's first four bytes for zstd magic headers that are at the start of a frame. We can make something like a (*File).IsCompressed() method, so it's compatible with compressed and non-compressed content.