New CLI v2: Need testers and feedback
HACKERALERT opened this issue · 28 comments
A lot of people have requested a better CLI, so here it is!
https://github.com/HACKERALERT/Picocrypt/tree/main/cli/v2/picocrypt
It supports multiple files, folders, and glob patterns, as well as paranoid mode and Reed-solomon for both encryption and decryption. It is much better than the relatively stubborn v1 CLI, and I hope that this will satisfy a lot of your needs. I wrote most of the CLI v2 code in one sitting, so please try it out and confirm that it works by leaving a comment. As well, please try different combinations and parameters to try to break the CLI and let me know of any bugs so that I can make it as stable as possible. Thanks :)
@HACKERALERT I've only discovered 1 issue. It is with extremely Slow decryption of files/folders encrypted using reed-solomon. Roughly writing 1MB of data per ~10 seconds and seems to get slower every second. Which does not occur through GUI. Issue persist when combing with paranoid mode as well. Encrypting with Reed-solomon works as usual. Like it was running for 5 minutes before finishing decrypting 70MB pcv
I am running on Linux, distro used is Kubuntu. Go latest version is 1.22.2.
Ah yes, I (temporarily) forced slow decode because the code structure doesn't allow for a "second pass" to actually decode the RS parity bytes if corruption is detected. I might add something like "picocrypt -[f]ix" that will cause RS to actually fix the errors, and by default go fast. Not sure if that made sense, but don't worry, I will fix this bug.
Also, stay tuned, I have cool progress bars working and will push shortly.
Progress bars added (I will fix the RS issue later)
To update:
go install github.com/HACKERALERT/Picocrypt/cli/v2/picocrypt@77aea6a
Progress bars work great.
I also noticed that the flags such as -p must be issued before that of the file/folder to encrypt. For e.g if I issue command "Picocrypt * -p" it will not use paranoid mode, Instead I must issue "Picocrypt -p *". All software I've used such as rsync, rclone...it doesn't matter where the flags are specified in the command. Is this intentional, or something you can address? Or at least output an error mentioning it?
I'm using the standard Go flag package to parse flags, which is what basically all Go CLI apps use, so it is probably well-known behaviour [https://stackoverflow.com/a/25249682]... that said, I don't think this is too big of an issue, right? The usage does show -p and -r before the args and most command line apps do use this as the default order.
Well, most people that use the CLI probably are more advanced anyway. Its mainly that most Linux users at least when working with the CLI are used to being able to place the flags anywhere on the line, and if they do this with Picocrypt it just won't be using the features there expecting.
If you can't address it, could you just error it out and throw a msg informing a user of this? Or is this also not possible with the Go's flag package? If you can't do this either, then just having it well-documented would have to suffice.
The only reason. I figured this out was when I added the -p flag at the end, and noticed encryption speed the same as default, it got me curious why it was faster than Paranoid mode in the GUI.
The Reed-Solomon issue should be fixed. To update:
go install github.com/HACKERALERT/Picocrypt/cli/v2/picocrypt@97bde62
To test, encrypt something with Reed-Solomon. Decrypt it, it should be fast. Then corrupt a few bits of the volume and try again. It should fail with an error message prompting you to use the -fix flag. Use that; it will be slow but it should fix the issues and the decrypted file(s) should match the original files.
Note to self: For some reason the progress bar when encrypting/decrypting stops at 98/99% and doesn't go away like it should.
Understood, thanks for the perspective. I've made it error out if you put -p -r or -fix (-f in a future commit) in the wrong place. To update:
go install github.com/HACKERALERT/Picocrypt/cli/v2/picocrypt@1b4998d
(typically go install ...@latest is enough, but with frequent commits it doesn't update fast enough so putting a commit manually forces it to repull)
I see that it does indeed error out now if flags are specified wrong, the reed Solomon does indeed decrypt at normal speeds now.
When attempting to recover corrupted file, it outputs error that the volume header is damaged. When using command: picocrypt -fix *
I did not even corrupt the area where the header is stored, as can be seen in a hex viewer like xxd.
I should note that the error: the volume header is damaged is the same when I try to decrypt the corrupted file normally, as well if I use the -fix flag. So it doesn't even instruct me to use -fix.
Uh I don't think picocrypt -fix * works, it should be picocrypt -fix volume.pcv
I changed the -fix to -f, to update:
go install github.com/HACKERALERT/Picocrypt/cli/v2/picocrypt@613ab82
I think the reason why you are getting the error is because you corrupted a contiguous block of bytes, probably more than 8 bytes. This is expected to error because RS is done in 128+8 configuration. In real life, bit rot typically picks random single bits to corrupt, so it is extremely unlikely to have a "large" sector of bytes wiped out, and if that did happen, it would probably be much more than 8 bytes and nothing can recover it.
So we only need to consider the practical possibilities. For reference, the header is the first 789 bytes. So:
picocrypt -p -r *
This will create a volume. Now decrypt it:
picocrypt encrypted.zip.pcv
This should be fast since no corruption occured yet. Now let's corrupt the 10th byte in the header:
f = open("encrypted.zip.pcv","rb+")
f.seek(10)
f.write(b"0")
f.close()Decrypt again:
picocrypt encrypted.zip.pcv
Again, it should be quick since the header is always RS encoded and reassembled. Now corrupt the data:
f = open("encrypted.zip.pcv","rb+")
f.seek(1000)
f.write(b"0")
f.seek(2000)
f.write(b"0")
f.seek(3000)
f.write(b"0")
f.close()Now decrypt:
picocrypt encrypted.zip.pcv
You should get the error and a message telling you to use '-f'. Use it:
picocrypt -f encrypted.zip.pcv
It should decrypt slowly but the output should be without any corruption.
Try this if you are able to and let me know if you can follow it through without error. If you can, then I think it demonstrates that everything is working, theoretically at least.
It worked perfectly by using the fopen function to corrupt the file instead. The issue probably was that I corrupted too much of the file for Reed Solomon to recover from.
Do you think the error stating that the volume header is damaged in the case of my example may be misleading if the file is just too badly corrupt (or the header itself is not actually damaged)? Someone may think that there's a bug, if its not reporting anything about it. Maybe by stating something like the volume may be too corrupt for recovery.
Thanks for the update. Could there be any ways to pass password as an argument like previously?
Also can I set output file name instead of encrypted.zip.pcv?
@CodeCracker-oss Good point, I've changed
"The volume header is damaged" -> "The volume header is irrecoverably damaged"
for if RS rebuilding fails and we know for sure the header is broken (and therefore the rest of the volume).
In the case of the volume itself being unable to recover via RS, first it will prompt -f, but use -f and it will throw an error as soon as one block of data is unrecoverable. So that should cover all cases.
@ntnguyen1234 I decided to prompt the user for the password in v2 so that the password isn't stored in the command history you can access in any shell by pressing the up arrow. Are you intending to write a shell script that uses the CLI? You have two choices, use v1 (which is legacy but I don't plan on removing from the repo and works just fine), or echo the password into v2. For the latter, I'm not a command line pro, but it's probably something like echo "pwd\npwd\n" > picocrypt * (@CodeCracker-oss maybe you know what it is?). It's not the cleanest, but it does work if you really need the extra features of v2.
As for the output file, I might add a -o flag. Still thinking about it.
@HACKERALERT When you ask me if I know what it is, are you referring to echoing the password into picocrypt? The proper way to echo a password on Linux into picocrypt would be (for e.g): echo "pwd" | picocrypt *
However picocrypt itself outputs an error stating "Password: Error reading password."
And my last thing to ask, is if you plan on adding a help page, like --help that will mention all flags one could use, and short description of them? Maybe even a version flag, so one could tell what version their running.
@CodeCracker-oss I might rewrite the CLIv2 so that I can clean up the flags and make them show upon running -h or without any parameters. It'll be a bit of effort but we'll see. A version flag is probably not necessary as I plan for the CLI to be decoupled from the GUI, so it would get updates whenever I make a fix or improvement.
@HACKERALERT It seems like its almost there though, the -p and -r are already mentioned. Maybe having the flags in a vertical list rather than horizontal would be easier to read but it would seem like a lot of extra work for you to rewrite it over this small concern, the -h flag wouldn't actually be necessary, as can currently just use no parameters.
Usage: picocrypt <item1> [<item2> ...]
Items: can be files, folders, or globs
Flags:
-f (decryption) attempt to fix corruption
-k (decryption) keep output unconditionally
-p (encryption) use paranoid mode
-r (encryption) encode with Reed-Solomon
Yup, I've decided to rewrite the CLI and I'm making some progress. Go adds -h and --help automatically so it's all good.
@ntnguyen1234 I decided to prompt the user for the password in v2 so that the password isn't stored in the command history you can access in any shell by pressing the up arrow. Are you intending to write a shell script that uses the CLI? You have two choices, use v1 (which is legacy but I don't plan on removing from the repo and works just fine), or echo the password into v2. For the latter, I'm not a command line pro, but it's probably something like
echo "pwd\npwd\n" > picocrypt *(@CodeCracker-oss maybe you know what it is?). It's not the cleanest, but it does work if you really need the extra features of v2.As for the output file, I might add a -o flag. Still thinking about it.
Thanks for your reply. Yeah I'm indeed using my python script to backup things daily and the commands run by python's subprocess are not logged in command history.
@ntnguyen1234 Thanks for the info. I don't plan to allow passing passwords via the CLI for the moment since users can continue using v1 for that purpose. I might add it in the future to v2, but for now, I'm focusing getting v2 stable with what I currently have implemented.
I understand. Thank you for your work 🙂
@CodeCracker-oss I completely rewrote the CLI, it should be super stable now and handle Ctrl+C exits properly. If you have time, give it a go and try to break it. Thanks!
go install github.com/HACKERALERT/Picocrypt/cli/v2/picocrypt@0a6e6bb
@HACKERALERT So far, so good. Haven't been able to break it, i've used all options and combinations. I noticed one small issue when the process exits the shell, it doesn't indent back down. I've attached a screenshot of my terminal for that.
The -k flag mentions to keep output unconditionally, can you explain what this means? The flag didn't appear to do anything during decryption, as I'm not sure what its supposed to do I can't really confirm if it works properly.
Funny because on my Windows, it does go back to a newline. I've made it print a newline so that should fix it.
go install github.com/HACKERALERT/Picocrypt/cli/v2/picocrypt@e330cfc34fb18cde1413413e096211957bd117a7
-k means keep corrupted output. I updated the flag description to better describe this. To test it, just make a standard (no -r or -p) volume, corrupt byte 1000 or something like that. Decrypt it and you'll see what I mean, use the -k flag to force keep output.
It does indeed print on a new line now. I've noticed a lot of software over the past on linux has suffered from this issue, I mean its not a deal breaker or anything, just a minor annoyance Never seen this issue from software I've used on windows though.
I have tested the -k, and I see what you mean now. Normally it doesn't keep corrupted output, but the --k does. So confirm that does work well.
Thank you for all your work on this 👍
I had just noticed when I tried the CTRL+C to cancel, that for this it does work fine as before but also does not print a line when exiting the shell. Same as before
Thanks for catching this, should be fixed now.
go install github.com/HACKERALERT/Picocrypt/cli/v2/picocrypt@3804a6c
Thanks for all of your help! The CLI would not be as polished as it is now without your scrutiny :)
With all that said, I think the CLI is in a fairly stable state now and this issue can be closed. Of course, if you discover any issues, bugs, or inconsistencies in the future, open an issue and we'll take it from there.
Thought I should note that Picocrypt CLI v2 appears to be just as stable as its Linux version for those who wish to use it on android via termux