configuration refactoring

[nihlaeth]

We were talking about it in the ssl pull request, and well, I did a thing... It's early stages right now, and the documentation looks atrocious, but I thought it might come in handy here. If you want any features added to make it more suitable for aenea, I'm all ears.

[calmofthestorm]

Sorry for the delay.

On the server: Anything is better than what we currently have. I really dislike the whole config.py setup, and the config.py.example business. I think it should be fairly straightforward to package the server as a proper, pip-able, package, which would mean the functionality to specify the config file as you've done in the link -- default, system-wide, user, etc. I don't have a strong opinion either way in terms of cli flags. It's 2017, so I don't care much about adding another dependency provided it can be easily and automatically installed. That said, I'm somewhat on the fence over what this adds compared to just reading ~/.aenea.yaml (or something in .local or wherever is appropriate; I'd have to refresh myself on that point), and falling back to sane default (and writing a config file). One argument would be multi-platform support, but currently the different servers for different platforms are wildly different. There's plugin support, but we could probably work something out, and honestly I'm not sure if anyone uses the plugin framework.

On the client: I'm a lot more negative here, because the config problem we need to solve is a lot dicier, and the code that implements it is mature and well tested.

First of all, there are several different configurations:

• There's the whole how-to-find-the-project-root business which is terribly complex and poorly documented. This is something I think you solve, though I would still need a way to do all the configuration out of an arbitrary location (VM shared folder), so we still need something like the config root setting.

• There's the per-grammar configuration. I'm especially adverse to changing this because I don't know how many users out there it would break. Could be zero, could be a lot. I'm not saying I'd never change anything that breaks backwards compatibility; only that I would need a compelling reason to do so.

• There's the current state configuration. This is both read and written, and needs to be watched (see configuration.py and its tests). Some users use this to drive multiple servers off one client. Additionally, it's how the static and dynamic grammar stuff works.

Assuming you implemented YAML support, we could potentially use something like this to handle the first two. However here I'm looking at mature, stable, and well-tested code that does the configuration, so I would need a compelling reason to move, regardless of who does the actual changes.

For the third, perhaps it makes sense to simply treat it as a separate system -- the files don't really ever need to be read/written by a human, just aenea and possibly user grammars/automation.

I guess I don't see moving to this as doing much to address configuration complexity -- I should document what we currently have better, and perhaps try to simplify the recursive load setup for config root.

Let me know if I'm missing something.

[nihlaeth]

First of all I want to be clear that I'm not saying you should definitely use user_config. I mostly opened this issue to get a discussion started on improving configuration. That said, I am personally very happy with user_config and would be happy to adapt it to suit the needs of aenea.

That said, I'm somewhat on the fence over what this adds compared to just reading ~/.aenea.yaml (or something in .local or wherever is appropriate; I'd have to refresh myself on that point), and falling back to sane default (and writing a config file).

The advantage of using that library over ~/.aenea.yaml is that you define options, validation, defaults and documentation in the same place, and it's exposed to the user in a readable way as well.

One argument would be multi-platform support, but currently the different servers for different platforms are wildly different.

If you want to make the server pip installable, I would recommend making one package that works on every operating system. How different are the configuration files for the different operating systems?

There's plugin support, but we could probably work something out, and honestly I'm not sure if anyone uses the plugin framework.

I've seen a plug-in that displays microphone state, but none other. Do plug-ins even use the configuration? If they do, wouldn't it be better to have them use their own configuration files?

There's the whole how-to-find-the-project-root business which is terribly complex and poorly documented. This is something I think you solve, though I would still need a way to do all the configuration out of an arbitrary location (VM shared folder), so we still need something like the config root setting.

The user_config.Config class allows you to overwrite the user and global paths. I would only overwrite the user path in this case, preserving the global path for system-wide setting fallbacks.

There's the per-grammar configuration. I'm especially adverse to changing this because I don't know how many users out there it would break. Could be zero, could be a lot. I'm not saying I'd never change anything that breaks backwards compatibility; only that I would need a compelling reason to do so.

Is this something that happens in configuration files, or with python code? In case of the latter, there's a good chance that using user_config wouldn't break anything at all, since it supports both attribute and dict-like access.

There's the current state configuration. This is both read and written, and needs to be watched (see configuration.py and its tests). Some users use this to drive multiple servers off one client. Additionally, it's how the static and dynamic grammar stuff works.

Good point, that falls clearly outside of the domain of user_config.

I personally rather don't document state files, and prefer users don't fiddle with them, only code. In that philosophy, static and dynamic grammar stuff absolutely belongs in a state file, but using multiple servers for one client belongs in permanent user configuration. Maybe it's an idea to re-factor that?

Assuming you implemented YAML support, we could potentially use something like this to handle the first two.

I plan on doing just that before the first beta release.

However here I'm looking at mature, stable, and well-tested code that does the configuration, so I would need a compelling reason to move, regardless of who does the actual changes.

I've got the well-tested part covered (>90% test coverage actually), the rest not so much. User count of 1 and still in alpha. I don't know of any other packages with similar functionality however.

You could always write it yourself, but that's more work and also untested and immature code then. The question maybe should be how much you want the multiple fallback options (command-line, global and user), especially since command line doesn't make much sense for the client. Or how much you like the interface that user_config exposes.

For the third, perhaps it makes sense to simply treat it as a separate system -- the files don't really ever need to be read/written by a human, just aenea and possibly user grammars/automation.

I agree, and I think moving away the multiple server configuration from there makes sense in any case.

I guess I don't see moving to this as doing much to address configuration complexity -- I should document what we currently have better, and perhaps try to simplify the recursive load setup for config root.

I disagree. user_config adds an abstraction layer, removing the need for you to define fallback and default logic. Moreover, I think that code using user_config is more readable, and that alone reduces complexity.

It also encourages you to document every option, which both documents your code and exposes said documentation to your end-user. Otherwise you end up writing documentation for the end user separately, and that often ends up being out of date or incomplete. Plus users often have a hard time finding it (or don't bother looking). this way, the documentation is inside the configuration file, and can hardly be missed.

I'm happy to help you refactor/document this whatever you choose to do.