importing pkg_resources fails
lithp opened this issue · 5 comments
I believe that pkg_resouces is simply incompatible with pyoxidizer, but I've only been playing with pyoxidizer for a few hours so maybe someone here knows better. Is there a way to get pkg_resources to work? It's used in quite a few places by the tool I'm trying to package up!
If I init a new project and then make two changes:
[[packaging_rule]]
type = "pip-install-simple"
package = "setuptools>=36.2.0"
[[embedded_python_run]]
mode = "eval"
code = "import pkg_resources"
Then pyoxidizer run gives me a fun error:
Traceback (most recent call last):
File "linecache", line 95, in updatecache
FileNotFoundError: [Errno 2] No such file or directory: 'pkg_resources._vendor.pyparsing'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "pkg_resources", line 84, in <module>
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
File "pkg_resources.extern", line 43, in load_module
File "pkg_resources._vendor.packaging.requirements", line 9, in <module>
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
File "pkg_resources.extern", line 43, in load_module
File "pkg_resources._vendor.pyparsing", line 4756, in <module>
File "pkg_resources._vendor.pyparsing", line 1284, in setParseAction
File "pkg_resources._vendor.pyparsing", line 1066, in _trim_arity
File "pkg_resources._vendor.pyparsing", line 1050, in extract_stack
File "traceback", line 211, in extract_stack
File "traceback", line 363, in extract
File "traceback", line 285, in line
File "linecache", line 16, in getline
File "linecache", line 47, in getlines
File "linecache", line 103, in updatecache
File "<frozen importlib._bootstrap_external>", line 566, in decode_source
AttributeError: 'memoryview' object has no attribute 'decode'
error: cargo run failed
Here's the relevant code:
It looks like python is trying to stat the file. The stat fails because pyoxidizer is like that. Then, something deep inside importlib is attempting to use the memoryview which pyoxidizer passes in and failing, probably because it expects straight bytes?
I hit the following today:
Traceback (most recent call last):
File "linecache", line 95, in updatecache
FileNotFoundError: [Errno 2] No such file or directory: 'pkg_resources._vendor.pyparsing'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "importlib.util", line 94, in find_spec
File "rdt", line 23, in <module>
File "Common.ET.envTracker", line 4, in <module>
File "slumber", line 1, in <module>
File "requests", line 112, in <module>
File "requests.utils", line 26, in <module>
File "requests._internal_utils", line 11, in <module>
File "requests.compat", line 29, in <module>
File "simplejson", line 114, in <module>
File "simplejson.decoder", line 8, in <module>
File "simplejson.scanner", line 11, in <module>
File "simplejson.scanner", line 7, in _import_c_make_scanner
File "simplejson._speedups", line 7, in <module>
File "simplejson._speedups", line 3, in __bootstrap__
File "pkg_resources", line 84, in <module>
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
File "pkg_resources.extern", line 43, in load_module
File "pkg_resources._vendor.packaging.requirements", line 9, in <module>
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
File "pkg_resources.extern", line 43, in load_module
File "pkg_resources._vendor.pyparsing", line 4756, in <module>
File "pkg_resources._vendor.pyparsing", line 1284, in setParseAction
File "pkg_resources._vendor.pyparsing", line 1066, in _trim_arity
File "pkg_resources._vendor.pyparsing", line 1050, in extract_stack
File "traceback", line 211, in extract_stack
File "traceback", line 363, in extract
File "traceback", line 285, in line
File "linecache", line 16, in getline
File "linecache", line 47, in getlines
File "linecache", line 103, in updatecache
File "<frozen importlib._bootstrap_external>", line 566, in decode_source
AttributeError: 'memoryview' object has no attribute 'decode'
Any suggestions or workarounds?
not using pkg_resources is not an option, if my app requires fs (pyfilesystem2) which in turn requires pkg_resources :(
AttributeError: 'memoryview' object has no attribute 'decode' is an interesting error. What's likely happening here is the code path to get the Python module source doesn't like the fact that we are using a memoryview to represent source instead of bytes. This is a side-effect of PyOxidizer's in-memory module importing.
The code that is failing in Python's standard library is:
import tokenize # To avoid bootstrap issues.
source_bytes_readline = _io.BytesIO(source_bytes).readline
encoding = tokenize.detect_encoding(source_bytes_readline)
newline_decoder = _io.IncrementalNewlineDecoder(None, True)
return newline_decoder.decode(source_bytes.decode(encoding[0]))
and source_bytes here is a memoryview (as opposed to bytes).
We should be able to work around this by having our get_source() call into decode_source() with a bytes or bytearray instead of memoryview.
In other news, it is unfortunate that Python doesn't have a standard library type that supports 0-copy and exposes .decode(). We have bytearray, which is close. But the constructor for bytearray copies data into a new buffer. This is really unfortunate.
I was able to import pkg_resources using the commit I just pushed.