Provide template filter and map capabilities

[JakobGM]

Motivation

Now that recursive directory compilation has been implemented, we should add capabilities for filtering which templates that are compiled, and specifying what to name the resulting target files.

After implementing this, you could simply point Astrality to your dotfiles directory (both as compilation source and target), and use astrality.yml solely as a context specification configuration. This makes the use of Astrality much more low-maintenance. Just create a new "xyz.template" file in a directory, and it will be automatically compiled to "xyz". I really like the idea of that!

I consider this a necessary requirement for the v.1.0 release, as v.1.0 should be capable enough to manage all your dotfiles.

The drawback with just pointing Astrality to your entire dotfiles directory tree is that you loose any kind of modularity when it comes to your dotfiles. There are no longer any requirements for compiling templates besides the broad requirements specified in the main dotfiles module. Perhaps we could support some specific ".astrality.yml" file which is parsed by Astrality and determines if the directory content should be compiled based on the "requires" functionality? Well, this is a seperate issue.

Goals

The end result should support:

• Only compile a subset of files within a directory, let's call it "filter".
• Deterministically rename template files, let's call it "map".

Compile action parameter name

We don't necessarily need to use the keywords "filter" and "rename", the best keywords will depend on which syntax we decide upon in the end.

Alternatives to "filter": "match", "only", and "regex".
Alternatives to "map": "rename" (but I'm pretty sure "rename" is the best one here).

Syntax proposals

Filter

Let's look at different filter syntaxes that could specify "only compile .template files":

regex filter:

compile:
  filter: '.+\.template'

glob filter:

compile:
  filter: '*.template'

extension map:

compile:
  extension: '.template'

The last one is the simplest one, but also the most restrictive.

Map

And different map syntaxes that specify "rename xyz.template files to xyz":

regex map:

compile:
  map: '(.+)\.template'

The syntax above only supports one capture group, and no reordering.

vim map:

compile:
  map: 's/(.+)\.template/\1/'

This syntax has the added benefit of that we can just point to the vim documentation for how to write such expressions. Although probably most potential users know vim/sed, it should not be taked as granted. The syntax is quite archane (I think).

arrow map:

compile:
  map: '(.+)\.template -> \1'

This looks prettier, but is unconventional.

There is also the possibility of combining "map" and "filter" into one single parameter, let's call it "filter-map". With other words, only compile templates which matches the rename regex, and use the regex capture group as the target file name. Although more succinct, it is more restrictive.

Caveats

What happens if your source directory contains non-template files. Should they be moved instead of compiled? And how can we detect such non-template files? We should keep this in mind when specifying the syntax.

[JakobGM]

@sshashank124 feedback from #9.

[...] I think the extension option might be too tailored for a specific use-case.

I'm a fan of "convention over configuration" as long as the convention makes sense (often the hardest part), and I think that Astrality should be quite opinionated. It drastically reduces development time when we don't have to support a billion different use-cases. But in this case I think I agree, it might be too specific.

Personally, I prepend t. to my templates to allow vim to determine the filetype better in some cases without having to manually specify # vim:ft=... for when the extension for a template is .template.

I hadn't even thought about that, great point! Well, that throws the extension filter syntax out of the equation...

Also, I'm not a fan of the regex solution since it might not be the most intuitive for some people. I'm thinking that if there is a dedicated directory for a set of files to compile, it shouldn't be necessary to name the template files with a .template extension in the first place. But, it'd be nice to have the rename option without cutting it out completely.

I completely agree that "compiling all files" is the reasonable default here. We wouldn't want users to set any more parameters than they need to do right now when they want to compile an entire directory. "filter" and "map" should be completely optional.

All in all, the best course would be to compile files with the same filename unless a rename/regex field is specified where the regex option's value corresponds to the following in a typical regex expression s/abc/def/g, but maybe a little more straightforward. Either way, I don't think we should specifically limit the option to only work on extensions using extension

Yes, so the blocker is deciding on which syntax to use. I would appreciate your feedback on the syntax proposals above, @sshashank124!

[sshashank124]

I'm fine with convention over configuration. I had just mentioned that because I was just thinking about my one t. template-naming case. I think allowing arbitrary regex expressions might be overkill. We could instead just go with something similar to your original example of anyprefix(.+)anysuffix or by globbing with anyprefix*anysuffix where the resulting file's name is just the captured group. Anything more complicated than that is just poor template naming imo. In the case that the regex match fails, it will just use the template name for the compiled file.

As for the copy vs compiling, I was considering opening another issue for a similar functionality request. Similar to compile, the user should also be able to specify copy or link or something similar where the file is either copied or symlinked without the need for templating. This could be useful in the cases of binary files or when the user wants to use astrality as a dot repo but not every file in the repo is necessarily a template. This would make it an improved version of a symlinker with the added option of also compiling specific files with templates.

When it comes to the folder compiling, it could be like you suggested. Anything that matches the regex/glob is compiled as a template, the rest are copied/symlinked

[JakobGM]

I agree with the bad template naming argument, if we assume users to use sensible template names, then the syntax could be simpler. It would make arbitrary regexes unecessary.

If we want to go down that route we have to decide between anyprefix(.+)anysuffix and anyprefix*anysuffix. Both "filter" and "rename" should use the same base syntax, at least. In isolation, I would use globs for filters, and regexes for renames, but we should not have both.

The glob syntax is probably the most UNIX-like syntax, while the regex makes the name capture much more explicit. Additionally, the regex syntax allows some additional tweaking, such as ignoring/including hidden files and stricter pattern matches (for those inclined).

Finally, the regex syntax will be marginally easier to implement.

Simpler regex syntax

Example compiling and renaming all ".template" files:

compile:
  source: $XDG_CONFIG_HOME
  target: $XDG_CONFIG_HOME
  filter: '.+\.template'
  rename: '(.+)\.template'

Example compiling all files, but only renaming ".template" files:

compile:
  source: $XDG_CONFIG_HOME
  target: $XDG_CONFIG_HOME
  rename: '(.+)\.template'

Example moving all files non-template files, but compile+rename template files:

compile:
  source: ~/.dotfiles/config
  target: $XDG_CONFIG_HOME
  filter: '.+\.template'
  rename: '(.+)\.template'
copy:
  source: ~/.dotfiles/config
  target: $XDG_CONFIG_HOME
  filter: '!(\.template)$'  # This is probably not a valid negated regex pattern :/

After having written up these examples, I realize that it becomes quite verbose, with all the escaping and so on...

Include/exclude syntax

What about using "include" and "exclude" instead? If the "include" regex contains a capture group, use that for the target name, else, keep the name. Exclude can be used to ignore files (and capture groups will not make sense in exclude patterns).

Compile all .template files, removing extension. Copy the rest:

compile:
  source: ~/.dotfiles/config
  target: $XDG_CONFIG_HOME
  include: '(.+)\.template$'
copy:
  source: ~/.dotfiles/config
  target: $XDG_CONFIG_HOME
  exclude: 'template$'  # or '^.+\.templates$' if you want to be more explicit.

I actually like this syntax the most at the moment, but it is a really hard decisision 🙄

Implicit copy/symlink syntax

This shouldn't be too hard to implement. The question if we would want to implicitly copy or symlink all files that do not match pattern, and compile+rename the files that match the pattern. If no pattern is provided, consider all files templates, as it is now.

Or if it should be specified explicitly with copy and symlink action types. Ugh, yet another difficult decision.

Again, we should consider the 80% usecase. I think the most common scenario is a bunch of files, where "*.template" should be compiled+renamed and the rest should be either symlinked or copied (perhaps as a boolean compile action flag?). What about the following syntax?

Compile all .template files, removing extension. Copy the rest:

compile:
  source: ~/.dotfiles/config
  target: $XDG_CONFIG_HOME
  templates: '(.+)\.template$'
  symlink: false

Compile all .template files, removing extension. Symlink the rest:

compile:
  source: ~/.dotfiles/config
  target: $XDG_CONFIG_HOME
  templates: '(.+)\.template$'
  symlink: true

With some default value for symlink, for the most normal use-case. Perhaps symlink: false as default is most consistent, since compiled templates are copied?

[sshashank124]

I like the include/exclude version too. For the copy/symlink, I think that if we end up doing copy/symlink for remaining files in a folder, we should also provide the ability to do that individually using a symlink block similar to a compile block

[JakobGM]

So if I understand you correctly, you think we should use the include/exclude syntax, but not implicitly copy/symlink all remaining files, but rather have users explicitly specify copy and symlink instructions in seperate copy and symlink action blocks?

Because I don't think it makes sense to have implicit copy/symlink, in addition to explicit copy and symlink actions. Well, perhaps that's not true, we could do:

compile:
  source: ~/.dotfiles/config
  target: $XDG_CONFIG_HOME
  include / included: '(.+)\.template$'
  non_templates / excluded: 'symlink' / 'copy' / 'nop'

With some reasonable default for non_templates. Then additionaly provide explicit copy and symlink actions for good measure. What do you think about that?

What about non_template vs excluded vs other alternatives? And nop might by too technical, perhaps ignore instead?

[sshashank124]

Follow-up to my comments in #20 , we should also provide separate individual options for copy and symlink. In response to the comment above, it would be something along the lines of the following. The exact naming can vary:

module/bspwm:
  on_startup:
    compile:
      source: bspwmrc.template
      target: $XDG_CONFIG_HOME/bspwm/bspwmrc
    copy:
      source: wallpaper.sh
      target: $HOME/.fehbg
    symlink:
      source: wallpaper.jpg
      target: $XDG_CONFIG_HOME/wallpapers/wallpaper.jpg
    stow:                 <----- this is the folder option
      source: scripts_folder
      target: $XDG_CONFIG_HOME/bin

[sshashank124]

The goal is that the user should have each of the 3 main options: compile, copy, and symlink available for use individually and then the hybrid option stow or whatever it is called will be for the "compiling a whole folder" option. The goal is to not also call it compile to keep the distinction between the 3 individual actions and then the possibility of combining them through a stow/folder option.

Thoughts?