Please revisit creating archive using a directory as a source
Closed this issue · 5 comments
./target/release/mlar create -p key.pub -o my_archive.mla /home/user/dtmfcopy3/
/home/loongson/redoxstuff/dtmfcopy3/
[!] Command ended with error: IOError(Os { code: 21, kind: Other, message: "Is a directory" })
The concept of accepting only specified files on the CLI is insufficient as there are true memory/environment constraints limiting the length of the CLI command line. Inherently this limits the number of files you may pass to your CLI to create the target .mla archive file.
As example of the imposed constraint of your tool. What if a directory holds a significant number of files you want to archive in an .mla file? For example there are perhaps millions of files, data files, png files, vector files, that you may want to archive. The current design of mlar would not allow for that amount of files to be archived within one .mla file because we cannot construct one CLI listing those millions of files since it would probably core dump or simply error out with the sheer number of files listed at the CLI. It is not recommended to simply expect all files to be listed within the CLI in order to use mlar. This approach is insufficient to handle large quantities of files within one .mla archive file. Archiving implies a large number of files to begin. It should be designed for such scenarios in mind in order to gain popularity or mlar will simply be ignored as an alternative to archive files.
For the sake of gaining more popularity and usage of mlar, please reconsider the intended user audience of mlar and their requirements. Users simply cannot specify millions of files on the CLI to create an archive. That is unreasonable.
It truly is necessary for your tool to walk the specified source directory path to archive whatever is within that directory path.
There are rust-based directory walkers available in fact they also walk the directories in parallel.
Thank you for listening.
Please have a look at this code which is the equivalent of "find blahdir -type f -print" but in rust.
https://github.com/omac777/walkdir/tree/justfiles_findtypef
I hope it will be useful to support the archive of target directories.
I'm not re-opening, but I would like to add clarification to this matter.
You mentioned using '-' a number of times. I'm not against this method for others and for flexibility.
I do prefer explicit using "input file containing a list files" or "input file containing a list of directories"
as they are a pre-determined and pre-defined set established for the reasons of auditability before even archiving them.
Your tool mentioned another alternative with MLA,
"after the first wave of insertions to the archive, add some more files directories".
although I appreciate that degree of flexibility, my intention would be to only use this as a last resort.
Once a list of files/directories has been provided to create the archive, that's it. That is sufficient for my requirements which is different from yours.
I do appreciate your perspective and all the hard-work and reflection in these matters. Thank you so much.
Thanks for your clarifications.
Your tool mentioned another alternative with MLA,
"after the first wave of insertions to the archive, add some more files directories".
I'm not sure to understand this part.
For now, once an archive has been produced, one cannot add any more files. I don't know if this behavior will change one day, as it might raise some difficulties in the layer implementation.