Ansible-style variable interpolation

[beasteers]

So this is a thought I've been having for a while, but I wanted to get it on paper. I realize that it's not trivial and I understand if there is resistance to the thought.

In ansible, you can use jinja2 templating to substitute variables into your config. Those variables could be overwritten either by another config or via command line.

Here's my thought:

On the python side:

config = confuse.Config() # no interpolation - same as now
config = confuse.Config(vars_from='vars') # use any vars relative to `vars` key
config = confuse.Config(vars_from='.') # use vars relative to root
config.set({'model_version': v4})

Now in the config you can do:

vars:
  model_version: v3
# or just (w/ vars_from='.')
model_version: v3

model:
  path: /path/to/model_{{ model_version }}.tf
  output: /output/path/embedding_{{ model_version }}.csv

upload:
  upload_file: '{{ model.output }}'

(In order to mimic ansible, the jinja would need to be evaluated when resolving the key, not when loading the config file, btw)

[jackwilsdon]

This sounds like something that would be built on top of confuse as opposed to something that would be built into it - confuse.Config could be extended to implement this in an application.

[sampsyo]

To be perfectly honest, I am also a little wary of taking on Ansible-esque variable replacement in strings… mainly because it's a deceptively complex thing to do. Not only do we need a template language (e.g., Jinja2), we also need to interpose on all strings read from YAML files (and perhaps even the non-strings, which would be harder), invent a way for references like model.output to get data from the configuration before we have finalized the data, potentially contend with infinite loops in cyclic references, report errors when trying to interpolate data of a type that can't easily be converted to strings, etc. That seems like a lot of headache!

Personally speaking, I also sometimes don't very much like using config systems that work like this. Sometimes I think it would be nicer if the application developer had provided a more domain-specific way to do what I wanted—for example, to generate unique filenames based on a specified model version—but instead I'm stuck kinda needing to "code it up myself" using configuration interpolation instead. Does that make sense?

We do have some (optional) support for interpolating environment variables (cf #59). Maybe some kind of opt-in, narrow version of interpolation like that would be more tractable?

My deepest apologies, by the way, for letting your very useful PRs languish for a couple of weeks—it has been a really busy time here. I promise to get to reviewing them real soon!

[beasteers]

I agree that it's probably better kept separate from the confuse core so I second the thought about having it as a separate package. 👍

And I understand that as a generic config mechanism that it adds too much complexity and I wouldn't want that enabled by default either.

I'm just working on a project where my team members are discussing using config files as a pseudo programming interface where they can configure blocks in a dynamic way, meaning that the core package developer would not have the level of control necessary to preempt the types of workarounds that you're talking about. So in some cases, people may be trying to 'code it up'.

To address the concerns posed:

• yes you would need to attempt parsing jinja on any strings
• you wouldn't need to on other types (though you would iterate lists/dicts and apply recursively)
• you wouldn't need to worry about it being "finalized" - that's the point of a ConfigView
• yes cyclic references would be a issue, but that comes with the territory

---

regarding the other PRs - no worries! I've also been kinda swamped so I'm fine without an extra review cycle atm 😄

[beasteers]

I got kinda curious so I took an hour to see how far I could get it and I managed to get something working relatively well (with a super simple implementation as well).

I took some code from ansible that does the ast parsing which is the only piece that is a bit more involved.

So apparently, with a tiiiny bit of monkey patching, it's possible! 😄🎉🎉
https://github.com/beasteers/confuse_jinja

[sampsyo]

Wow; that's super cool! Nice work, and snazzy monkey-patching!