Add preset `run/batch/submit/alloc` flags for simplifying rabbit submission
Opened this issue · 5 comments
The normal way to ask for rabbit storage for a job is to add the --setattr=dw="#DW jobdw type=xfs capacity=100GiB name=myname" option to flux submit/run/batch/alloc but I was discussing today with @behlendorf whether there could be some simpler preset flags with configurable meanings. What we discussed was something like have flags like --small-xfs or --large-lustre that simply add a #DW string to the user's job, where that string could be configured at the system level, maybe in a config file somewhere. This would just be to simplify things for most users, advanced users could still set the dw attribute directly.
We don't currently have a way to add arbitrary options to the submission cli commands.
However, you could support these custom strings in jobspec attributes if that's acceptable, e.g. --setattr=dw=small-xfs or --setattr=dw=large-lustre. If it helps, there's a proposal to add a -S short option for --setattr so users could use -S dw=small-xfs.
You could also support --setattr=small-xfs, but this seems a bit dangerous since the datawarp hints/flags are not namespaced off from other attributes in jobspec.
This could possibly be handled with a frobnicator plugin.
Alternately, we have discussed adding plugin support to the Jobspec class and submission cli commands. This could possibly allow addition of new options to these commands for cases like this, though I'd want to think about that carefully. In that case I'd argue instead of generic options like --small-xfs and --large-lustre something like --dw=TYPE would be better (more easily extensible and checked, less likely to conflict with other plugin-provided options). This is perhaps academic since no support for adding options has been discussed or designed at this point.
However, you could support these custom strings in jobspec attributes if that's acceptable, e.g. --setattr=dw=small-xfs or --setattr=dw=large-lustre.
That sounds reasonable to me. I'll check with Brian.
Presumably I could add a new config table that specifies what special shorthand strings are accepted and what their meaning is (i.e. what #DW blah blah they correspond to)? I'm 90% sure I know how to do that.
Yeah, good idea. That would allow admins to customize the list.
If it helps, there's a proposal to add a -S short option for --setattr so users could use -S dw=small-xfs.
I do think this would be a big help.