importlib.resource

[abuckenheimer]

I'm piecing together a few parts of the documentation here so this may be a misunderstanding on my part but I'm having trouble getting the importlib.resource to act like I expect. Heres a trivial example I built (diff from a pyoxidizer init resource skeleton)

diff --git a/pyoxidizer.toml b/pyoxidizer.toml
index b401b2d..70297a4 100644
--- a/pyoxidizer.toml
+++ b/pyoxidizer.toml
@@ -78,10 +78,11 @@ type = "write-license-files"
 path = ""
 
 # Package .py files discovered in a local directory.
-# [[packaging_rule]]
-# type = "package-root"
-# path = "./src"
-# packages = ["foo", "bar"]
+[[packaging_rule]]
+type = "package-root"
+path = "./src"
+packages = ["foo"]
+include_source = true
 
 # Package things from a populated virtualenv.
 # [[packaging_rule]]
@@ -99,15 +100,16 @@ path = ""
 # Python interpreter is invoked, this section is not relevant.
 [[embedded_python_run]]
 # Run an interactive Python interpreter.
-mode = "repl"
+# mode = "repl"
 
 # Import a Python module and run it.
 #mode = "module"
 #module = "mypackage.__main__"
 
 # Evaluate some Python code.
-# mode = "eval"
-# code = "from mypackage import main; main()"
+mode = "eval"
+# code = "from foo import show_resources; show_resources('foo')"
+code = "from foo import show_resources; show_resources()"
 
 # Keeps track of which version of PyOxidizer is managing this project.
 # THIS SECTION IS MANAGED BY PYOXIDIZER AND SHOULD NOT BE CHANGED BY PEOPLE.
diff --git a/src/foo/__init__.py b/src/foo/__init__.py
new file mode 100644
index 0000000..6b3e5f5
--- /dev/null
+++ b/src/foo/__init__.py
@@ -0,0 +1,6 @@
+from importlib.resources import contents, read_text
+
+def show_resources(pkg=__package__):
+    print("package:", pkg)
+    print("contents:", contents(pkg))
+    print("text:\n", read_text(pkg, "__init__.py"))

so when I pyoxidizer run this I'd expect the following:

package: foo
contents: <list_iterator object at 0x7fab342760b8>
text:
from importlib.resources import contents, read_text

def show_resources(pkg=__package__):
    print("package:", pkg)
    print("contents:", contents(pkg))
    print("text:\n", read_text(pkg, "__init__.py"))

instead I get

package: 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "foo", line 5, in show_resources
  File "importlib.resources", line 248, in contents
  File "importlib.resources", line 47, in _get_package
  File "importlib", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
  File "<frozen importlib._bootstrap>", line 942, in _sanity_check
ValueError: Empty module name
error: cargo run failed

so package doesn't get set but maybe this is a known issue, but even if I specify the package in the embedded_python_run code = "from foo import show_resources; show_resources('foo')" above I still can't list the contents of the package

package: foo
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "foo", line 5, in show_resources
  File "importlib.resources", line 248, in contents
  File "importlib.resources", line 49, in _get_package
TypeError: 'foo' is not a package

which is also weird because its not consistent with another time I tried this where I could successfully call contents but it would return an empty tuple where as I would expect it to contain __init__.py. Apologies if I'm missing something obvious here, cool project though!

$ pyoxidizer -V
PyOxidizer 0.1.2
$ rustc --version
rustc 1.35.0 (3c235d560 2019-05-20)

[indygreg]

I would not at all be surprised if there were bugs in the behavior of the module resources APIs! There are a few issues to unravel in here. I'll definitely look at this in more detail in the days ahead unless someone beats me to it.

Thank you for reporting your experience!

[indygreg]

Thank you for the detailed reproduction case.

It looks like the behavior of importlib.resources.contents() and importlib.resources.read_text() are what you'd expect given a package name of ''. So that's good.

The underlying problem appears to be that __package__ is ''. It should be foo in this case. So if we fix __package__, importlib.resources should just work.

Lemme see about coding up a fix...

[indygreg]

OK, things should now behave properly on the main branch. Here is the new output from your test case:

package: foo
contents: []
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "foo", line 6, in show_resources
  File "importlib.resources", line 169, in read_text
  File "importlib.resources", line 125, in open_text
FileNotFoundError: resource not found

I think this is arguably correct behavior. It is slightly different from what you get from the file-based importer because .py files aren't registered as resources by default with the in-memory importer: only non-module files are registered as resources. The file-based importer does a blind os.list() and open() to load things.

If you want to pick up Python module files as resources (it would be a legitimate feature request), we could make that a configurable setting via the packaging rules. Please file a new issue for that if wanted.

[abuckenheimer]

thanks for taking a look! will open a new issue