jc -h and magic mode not working without xmltodict

Question

jc -h and magic mode not working without xmltodict

BubuOT opened this issue 9 months ago · 14 comments

The pygments and ruamel.yaml dependencies are optional, not installing them (we use jc on an embedded device build with buildroot) just disables certain functionality of jc, this is great for us as we don't need to install dependencies for functionality we never use.

For xmltodict this in theory works just the same, not installing it disables the xmltodict parser. The rest of jc keeps working with 2 notable exceptions:

jc -h refuses to work
jc <command> (I think it's called magic mode) also doesn't work

Both fail with this error:

jc:  Error - Exit due to unexpected error:
             LibraryNotInstalled: The xmltodict library is not installed.

Would be nice if global cli functionality would be unaffected by the dependency of a single parser. :-)

Answer 1 · 2024-02-08T17:14:11.000Z

Thanks for reporting this! I'll look into this right away. I'm not sure if this will add any more information, but could you please run jc -hdd? Thanks!

Answer 2 · 2024-02-08T17:15:50.000Z

Sure!

$ jc -hdd
LibraryNotInstalled
Python 3.11.7: /usr/bin/python
Thu Feb  8 18:14:25 2024

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /bin/jc in <module>()
   23         if entry_point.group == group and entry_point.name == name
   24     )
   25     return next(matches).load()
   26 
   27 
   28 globals().setdefault('load_entry_point', importlib_load_entry_point)
   29 
   30 
   31 if __name__ == '__main__':
   32     sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
   33     sys.exit(load_entry_point('jc==1.23.6', 'console_scripts', 'jc')())
sys = <module 'sys' (built-in)>
sys.exit = <built-in function exit>
load_entry_point = <function importlib_load_entry_point>

 /usr/lib/python3.11/site-packages/jc/cli.py in main()


 /usr/lib/python3.11/site-packages/jc/cli.py in run(self=<jc.cli.JcCli object>)


 /usr/lib/python3.11/site-packages/jc/cli.py in _run(self=<jc.cli.JcCli object>)


 /usr/lib/python3.11/site-packages/jc/cli.py in help_doc(self=<jc.cli.JcCli object>)


 /usr/lib/python3.11/site-packages/jc/cli.py in helptext(self=<jc.cli.JcCli object>)


 /usr/lib/python3.11/site-packages/jc/cli.py in parsers_text(self=<jc.cli.JcCli object>)


 /usr/lib/python3.11/site-packages/jc/lib.py in all_parser_info(documentation=False, show_hidden=False, show_deprecated=False)


 /usr/lib/python3.11/site-packages/jc/lib.py in _get_parser(parser_mod_name='xml')


 /usr/lib/python3.11/importlib/__init__.py in import_module(name='jc.parsers.xml', package=None)


 /home/deploy/<frozen importlib._bootstrap> in _gcd_import(name='jc.parsers.xml', package=None, level=0)


 /home/deploy/<frozen importlib._bootstrap> in _find_and_load(name='jc.parsers.xml', import_=<function _gcd_import>)


 /home/deploy/<frozen importlib._bootstrap> in _find_and_load_unlocked(name='jc.parsers.xml', import_=<function _gcd_import>)


 /home/deploy/<frozen importlib._bootstrap> in _load_unlocked(spec=ModuleSpec(name='jc.parsers.xml', loader=<_froze...lib/python3.11/site-packages/jc/parsers/xml.pyc'))


 /home/deploy/<frozen importlib._bootstrap_external> in exec_module(self=<_frozen_importlib_external.SourcelessFileLoader object>, module=<module 'jc.parsers.xml' from '/usr/lib/python3.11/site-packages/jc/parsers/xml.pyc'>)


 /home/deploy/<frozen importlib._bootstrap> in _call_with_frames_removed(f=<built-in function exec>, *args=(<code object <module> at 0x7f8d8ed3e0, file "/us...hon3.11/site-packages/jc/parsers/xml.py", line 1>, {'LibraryNotInstalled': <class 'jc.exceptions.LibraryNotInstalled'>, '__builtins__': {'ArithmeticError': <class 'ArithmeticError'>, 'AssertionError': <class 'AssertionError'>, 'AttributeError': <class 'AttributeError'>, 'BaseException': <class 'BaseException'>, 'BaseExceptionGroup': <class 'BaseExceptionGroup'>, 'BlockingIOError': <class 'BlockingIOError'>, 'BrokenPipeError': <class 'BrokenPipeError'>, 'BufferError': <class 'BufferError'>, 'BytesWarning': <class 'BytesWarning'>, 'ChildProcessError': <class 'ChildProcessError'>, ...}, '__cached__': '/usr/lib/python3.11/site-packages/jc/parsers/xml.pyc', '__doc__': 'jc - JSON Convert `XML` file parser\n\nThis parser...     "YEAR": "1988"\n          },\n      ...\n    }\n', '__file__': '/usr/lib/python3.11/site-packages/jc/parsers/xml.pyc', '__loader__': <_frozen_importlib_external.SourcelessFileLoader object>, '__name__': 'jc.parsers.xml', '__package__': 'jc.parsers', '__spec__': ModuleSpec(name='jc.parsers.xml', loader=<_froze...lib/python3.11/site-packages/jc/parsers/xml.pyc'), 'jc': <module 'jc' from '/usr/lib/python3.11/site-packages/jc/__init__.pyc'>}), **kwds={})


 /usr/lib/python3.11/site-packages/jc/parsers/xml.py in <module>()

LibraryNotInstalled: The xmltodict library is not installed.
    __cause__ = None
    __class__ = <class 'jc.exceptions.LibraryNotInstalled'>
    __context__ = ModuleNotFoundError("No module named 'xmltodict'")
    __delattr__ = <method-wrapper '__delattr__' of LibraryNotInstalled object>
    __dict__ = {}
    __dir__ = <built-in method __dir__ of LibraryNotInstalled object>
    __doc__ = None
    __eq__ = <method-wrapper '__eq__' of LibraryNotInstalled object>
    __format__ = <built-in method __format__ of LibraryNotInstalled object>
    __ge__ = <method-wrapper '__ge__' of LibraryNotInstalled object>
    __getattribute__ = <method-wrapper '__getattribute__' of LibraryNotInstalled object>
    __getstate__ = <built-in method __getstate__ of LibraryNotInstalled object>
    __gt__ = <method-wrapper '__gt__' of LibraryNotInstalled object>
    __hash__ = <method-wrapper '__hash__' of LibraryNotInstalled object>
    __init__ = <method-wrapper '__init__' of LibraryNotInstalled object>
    __init_subclass__ = <built-in method __init_subclass__ of type object>
    __le__ = <method-wrapper '__le__' of LibraryNotInstalled object>
    __lt__ = <method-wrapper '__lt__' of LibraryNotInstalled object>
    __module__ = 'jc.exceptions'
    __ne__ = <method-wrapper '__ne__' of LibraryNotInstalled object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of LibraryNotInstalled object>
    __reduce_ex__ = <built-in method __reduce_ex__ of LibraryNotInstalled object>
    __repr__ = <method-wrapper '__repr__' of LibraryNotInstalled object>
    __setattr__ = <method-wrapper '__setattr__' of LibraryNotInstalled object>
    __setstate__ = <built-in method __setstate__ of LibraryNotInstalled object>
    __sizeof__ = <built-in method __sizeof__ of LibraryNotInstalled object>
    __str__ = <method-wrapper '__str__' of LibraryNotInstalled object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __suppress_context__ = False
    __traceback__ = <traceback object>
    __weakref__ = None
    add_note = <built-in method add_note of LibraryNotInstalled object>
    args = ('The xmltodict library is not installed.',)
    with_traceback = <built-in method with_traceback of LibraryNotInstalled object>

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/jc/parsers/xml.py", line 77, in <module>
ModuleNotFoundError: No module named 'xmltodict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bin/jc", line 33, in <module>
    sys.exit(load_entry_point('jc==1.23.6', 'console_scripts', 'jc')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/jc/cli.py", line 891, in main
  File "/usr/lib/python3.11/site-packages/jc/cli.py", line 873, in run
  File "/usr/lib/python3.11/site-packages/jc/cli.py", line 808, in _run
  File "/usr/lib/python3.11/site-packages/jc/cli.py", line 322, in help_doc
  File "/usr/lib/python3.11/site-packages/jc/cli.py", line 287, in helptext
  File "/usr/lib/python3.11/site-packages/jc/cli.py", line 204, in parsers_text
  File "/usr/lib/python3.11/site-packages/jc/lib.py", line 567, in all_parser_info
  File "/usr/lib/python3.11/site-packages/jc/lib.py", line 273, in _get_parser
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.11/site-packages/jc/parsers/xml.py", line 79, in <module>
jc.exceptions.LibraryNotInstalled: The xmltodict library is not installed.

Answer 3 · 2024-02-08T17:18:22.000Z

Thanks - also, what version of jc are you running? (jc -v)

Reason I ask is that there was a change to the get_parser function in the latest release (v1.25.0) and I'm not sure if it would change the behavior. I can test on my side as well.

Answer 4 · 2024-02-08T17:20:29.000Z

The above output is from 1.23.6, but I reproduced this error in a venv with 1.25 on my system as well. (pip uninstall xmltodict after installing jc)

Answer 5 · 2024-02-08T17:54:49.000Z

I'm working on a fix - there are two issues:

The xml parser should probably import the library within the parse() function just as the yaml parser does
As a higher-level safety precaution, I can enclose the parser import in a try/except and insert a dummy "disabled" parser in case a parser does not correctly implement the import as above.

I can probably add some tests for this case.

I'll let you know when I have this ready to test. I can get this in v1.25.1 later today.

Answer 6 · 2024-02-08T18:05:01.000Z

There's one more case I found (which is very likely irrelevant for non-embedded use-cases):

the plist parser imports plistlib
plistlib depends on the python core xml module
the xml module is an optional component in buildroot (because it depends on libexpat)

It sounds like your second point would automatically also cover this issue of running under a python installation without the xml module present. Just wanted to make you aware of this additional data point :).

Answer 7 · 2024-02-08T18:35:16.000Z

Good call - I'll look into that as well. Thanks!

Answer 8 · 2024-02-08T19:05:10.000Z

Would you be against a STDERR warning message like the following when jc -h, jc -a, or the magic syntax is used? Unless I make larger code changes this warning would not be able to be quieted with -q:

jc:  Warning - "xml" parser disabled due to import error.

Answer 9 · 2024-02-08T19:11:43.000Z

A warning on stderr sounds fine to me.

Answer 10 · 2024-02-08T21:26:32.000Z

I found another issue that doesn't seem to be affecting anything at the moment but is not a good idea... the xml parser is shadowing the xml library when the plist parser is importing plistlib. This only seems to happen when calling the parser directly from the command line, but may also be affecting my ability to write a good test.

Accidentally got myself into a hairy situation here and it's surprising it's actually working today. I'm going to need to think about this a bit more...

Answer 11 · 2024-02-10T01:37:40.000Z

I have a fix working locally here - just working on creating some tests while the libraries are uninstalled.

Answer 12 · 2024-02-10T19:26:02.000Z

I have pushed the fix to the dev branch. If you can test let me know. I added the capability to test with and without the optional libraries installed. A broken parser (no matter the reason) should no longer cause a problem.

Answer 13 · 2024-02-12T11:09:28.000Z

Just tested now, looking good! :-)

Answer 14 · 2024-02-13T00:55:11.000Z

fix in v1.25.1