Default config not loaded when code deployed as egg package

[Marakai]

I use a default_config.yaml in my package, which is then deployed with setuptools as egg (not wheel yet).

When deployed that way with

python setup.py install

and the resulting code sitting in .../site-packages as *.egg, running my code results in it failing to read the default settings from the config file.

However, when installing the same package with

python setup.py sdist
pip install dist/package.tar.gz

where the code sits as an expanded directory in the "old way", it find the file and reads the settings fine.

What makes this frustrating to debug is that it also works fine in the IDE (Pycharm) so tracing and single-stepping shows nothing wrong.

In the egg, using good old print() statements, such as

config = confuse.Configuration('app', __name__)
print(config)
print(config.sources)

shows

[ConfigSource(<super: <class 'ConfigSource'>, <YamlSource object>>, '/home/username/.config/app/config.yaml', False), ConfigSource(<super: <class 'ConfigSource'>, <YamlSource object>>, '/home/username/.pyenv/versions/3.9.5/lib/python3.9/site-packages/app-0.8.28-py3.9.egg/app/config_default.yaml', True)]
ConfigSource(<super: <class 'ConfigSource'>, <YamlSource object>>, '/home/username/.config/app/config.yaml', False)
None
ConfigSource(<super: <class 'ConfigSource'>, <YamlSource object>>, '/home/username/.pyenv/versions/3.9.5/lib/python3.9/site-packages/app-0.8.28-py3.9.egg/app/config_default.yaml', True)
None

whereas when installing via expanded package, it shows

[ConfigSource(<super: <class 'ConfigSource'>, <YamlSource object>>, '/home/username/.config/app/config.yaml', False), ConfigSource(<super: <class 'ConfigSource'>, <YamlSource object>>, '/home/username/.pyenv/versions/3.9.5/lib/python3.9/site-packages/app/config_default.yaml', True)]
ConfigSource(<super: <class 'ConfigSource'>, <YamlSource object>>, '/home/username/.config/app/config.yaml', False)
None
ConfigSource(<super: <class 'ConfigSource'>, <YamlSource object>>, '/home/username/.pyenv/versions/3.9.5/lib/python3.9/site-packages/app/config_default.yaml', True)
OrderedDict([('StackName', 'mydefaultstack'), ('Repository', 'mydefaultrepo'), ('GitSource', 'github'), ('Branch', 'master'), ('Artifact', 'artifact.zip'), ('BuildSpec', 'buildspec.yaml'), ('Postbuild', False), ('Deployment', False)])

My setup.py has the boilerplate

packages=setuptools.find_packages(),
package_data={
    '': ['*.yaml', 'data/*.yaml'],
},

That points to loader having issues with packages in compressed/binary form, but maddening to pinpoint.

[sampsyo]

Hello! Indeed; eggs make the use of "auxiliary" (non-Python) files from Python code difficult and complicated. I honestly don't know the reliable way to make this work every time—it requires using a custom setuptools API, I think.

Eggs are falling out of favor these days… is there a particular reason why you need to use them, as opposed to wheels?

[Marakai]

Inertia, existing deploy templates, mostly, I suppose. If it's nothing with the packaged format, where it would occur with both egg and wheel, it may be a good excuse to move into the current century. ;)

[sampsyo]

Well, wheels are generally not imported directly from the zip:
https://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

…which is one of the reasons they've gained some traction.

I definitely have no objection to adding support for in-package defaults from zip files, etc., but I honestly just don't know the right/reliable way to go about it. Probably pkgutil.get_data, but I'm not sure that works everywhere? Could be worth a shot, but it would also make things a bit more complicated internally because we couldn't track all sources as coming from files on the filesystem.

[Marakai]

Just been yelled at that most if not all of our containers use Alpine. Hence no wheel. :-|

I have to support multi-OS (Win, Mac, Linux) plus a plethora of Docker containers and their respective OS. I suppose I can go multi-format with a pure source dist where necessary and wheel where possible. Right there is, of course, one reason for the inertia...

[Marakai]

Going to try and join the year 2021 and finally go the binary route. I'll look into something like cibuildwheel to hopefully support most platforms - and keep a pure sdist around for the Alpine folks. As it's not a confuse issue, should rightfully close this. Cheers!

[Marakai]

Quick comment on closed ticket: migrating to Wheel (plus original source distribution for backwards) compatibility went smoothly. Default config correctly read. Now for the proselytising...

[sampsyo]

Excellent!