Add some kind of --path-type [exact|glob] for delete
Opened this issue · 10 comments
ATM we have that option for download command.
I am yet not sure how CLI should look like given that we take target file paths or URIs. I guess we could add this option and then treat local paths as globs to be expanded by python locally and on the server. For URLs we would need some custom URI schema to indicate that it is a glob for a particular dandiset (e.g. dandi-glob://<instance name>/<dandiset id>[@<version>][/<glob>])
WDYT @jwodder ?
Prompted by
although not yet known if really needed in that case.
ah, actually in https://dandi.readthedocs.io/en/latest/ref/urls.html#resource-ids we already specify that if there is such an option and =glob then we have dandi://<instance name>/<dandiset id>[@<version>][/<glob>] so I guess the path is "clear" right?
I guess we could add this option and then treat local paths as globs to be expanded by python locally and on the server.
A local glob path should be expanded by the shell into the full list of paths, at which point dandi delete shouldn't need to do anything special.
I guess we could add this option and then treat local paths as globs to be expanded by python locally and on the server.
A local glob path should be expanded by the shell into the full list of paths, at which point
dandi deleteshouldn't need to do anything special.
in case of local path specification -- do we remove only "matching" filenames or if e.g. I specify to remove a directory, would I remove all assets (which might be more or less than I have locally) from that "directory" on the server? If it is just 1-to-1 match to local files in the folder, then indeed we could keep it consistent and pretty much ignore that option for local paths. If not -- then it is a matter of passing glob not expanded (quote it) and match in code locally and on server.
@yarikoptic If a local directory path is passed to dandi delete, it deletes from the server each asset whose path starts with the given directory path; the local directory contents are ignored.
ok, in case of --path-type=glob let's use provided URL as a glob (per comment above) and if not a URL (ie local path) - using glob.glob(PATH, recursive=True) locally and delete matching assets (or folders if glob ended with /) locally and remotely.
if not a URL (ie local path) - using
glob.glob(PATH, recursive=True)locally
Why wouldn't we just let globs be expanded by the user's shell?
and delete matching assets (or folders if glob ended with
/) locally and remotely.
dandi delete currently doesn't delete anything on the local side. Doing so would be a considerable change that, if desired, should be submitted as a separate issue.
if not a URL (ie local path) - using
glob.glob(PATH, recursive=True)locallyWhy wouldn't we just let globs be expanded by the user's shell?
for that you do not need any --path-type or in other words it is what you get if use --path-type exact. Stating --path-type glob would mean that user is specifying a glob explicitly (and better quotes so shell does not expand), and possibly benefits from local shell not having recursive glob matching at all. I think it would be nice to just be able to say delete --path-type glob '*_entity-bad*' and have all assets with _entity-bad be removed locally/remotely.
and delete matching assets (or folders if glob ended with
/) locally and remotely.
dandi deletecurrently doesn't delete anything on the local side. Doing so would be a considerable change that, if desired, should be submitted as a separate issue.
frankly I didn't realize that! then let's forget about "local" aspects in my above phrases. Should also make implementation easier, and we would file an issue if some user desires it is worth it ;-)
if not a URL (ie local path) - using
glob.glob(PATH, recursive=True)locally
As I stated above, local (non-URL) paths passed to dandi delete are only matched against server paths; the only role of the local Dandiset is in ensuring the paths are relative to the Dandiset root. If we want dandi delete to handle globbing of local/file/non-URL paths, shouldn't the globbing happen against server paths rather than local paths for consistency?
Also, do we want to support local path arguments that contain a glob in the Dandiset path portion, e.g., dandi delete /home/user/00000*/*.nwb when run inside a Dandiset at /home/user/000001? That would be tricky.
If we want
dandi deleteto handle globbing of local/file/non-URL paths, shouldn't the globbing happen against server paths rather than local paths for consistency?
yes, globbing should happen against server paths, likely just by passing the glob=True to assets listing endpoint.
Also, do we want to support local path arguments that contain a glob in the Dandiset path portion, e.g.,
dandi delete /home/user/00000*/*.nwbwhen run inside a Dandiset at/home/user/000001? That would be tricky.
Then let's do no such thing -- just state that in case of =glob passed to server as is and assumed from the top of the dandiset.