Reading compressed inventory from stdin results in decoding error
Closed this issue ยท 12 comments
Brief description
Trying to display an inventory in the terminal, by curling it and piping it to sphobjinv convert:
curl https://pip.pypa.io/en/latest/objects.inv -o- | sphobjinv convert plain - -
Expected behavior
sphobjinv should read stdin, try to decode it, but catch exceptions and consider it compressed.
Actual behavior
sphobjinv doesn't catch the UnicodeDecodeError.
To reproduce
% curl https://pip.pypa.io/en/latest/objects.inv -o- | sphobjinv convert plain - -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 9485 100 9485 0 0 14280 0 --:--:-- --:--:-- --:--:-- 14306
Traceback (most recent call last):
File "/home/pawamoy/.local/bin/sphobjinv", line 8, in <module>
sys.exit(main())
File "/home/pawamoy/.local/pipx/venvs/sphobjinv/lib/python3.10/site-packages/sphobjinv/cli/core.py", line 88, in main
inv = inv_stdin(params)
File "/home/pawamoy/.local/pipx/venvs/sphobjinv/lib/python3.10/site-packages/sphobjinv/cli/load.py", line 245, in inv_stdin
data = sys.stdin.read()
File "/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.10.8/lib/python3.10/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 119: invalid continuation byte
System information
- Device: Computer?
- OS: ArchLabs
Python environment
Python
3.11
Libraries
% pipx runpip sphobjinv freeze
attrs==22.2.0
certifi==2022.12.7
jsonschema==4.17.3
pyrsistent==0.19.3
sphobjinv==2.3.1
Additional context
Passing a temporary file descriptor to the command does not work either:
% sphobjinv convert plain <(curl -s https://pip.pypa.io/en/latest/objects.inv -o-) -
Error while parsing input file path:
FileNotFoundError: Indicated path is not a valid file
I looked into this a bit when I implemented the stdin stream capability, and something about running a compressed inventory through a pipe appears to mangle it---no matter what I tried, I couldn't get it to recognize correctly.
I'm guessing it has something to do with character encoding, as AFAIK the *nix piping system expects to be passing plaintext, not bytestreams. There could be an env variable out there you can set to change the encoding behavior (a "raw" pipe mode?), in which case it might be workable.
I didn't want to put the work in to figure that out until I was sure it wasn't YAGNI. Which--now, it's not. ๐ฌ
If you have any thoughts about things to try, please let me know!
Although this makes me think I'm wrong about this, in which case I'm not sure what went wrong.
Maybe at the *nix/Python interface....
Oh, I see, thanks for the prompt reply!
It seems hard to get right indeed, so I'll just tell you that you can close this, because the workaround is so easy: just actually download the inv file locally, then run sphobjinv.
I think you also have tickets open for accepting URLs, that would solve the issue as well, and provide an even better UX (no pipe needed, more cross-platform, etc) ๐
Hm. The feature is already there for pulling inventories directly from URLs.
Did you try:
% sphobjinv convert plain -u https://pip.pypa.io/en/latest/objects.inv -
?
Ah, my bad, I glanced at the issues and the usage (-h
) but didn't understand this was possible. Well that's exactly what I needed haha, thanks!
Aha, I need to add to the base help message (sphobjinv -h
) to indicate that there's additional help for each subcommand (sphobjinv convert -h
and sphobjinv suggest -h
).
It's already the case ๐
% sphobjinv -h
usage: sphobjinv [-h] [-v] {convert,suggest} ...
Format conversion for and introspection of intersphinx 'objects.inv' files.
options:
-h, --help show this help message and exit
-v, --version Print package version & other info
Subcommands:
{convert,suggest} Execution mode. Type 'sphobjinv [mode] -h' for more information on available options. Mode names can be abbreviated to their first two letters.
convert (co) Convert intersphinx inventory to zlib-compressed, plaintext, or JSON formats.
suggest (su) Fuzzy-search intersphinx inventory for desired object(s).
I had just missed the -u
option in the help for the convert subcommand ๐
Oh hey! So it is. I scanned my own help message too quickly... ๐
Though -- since you missed it, probably means I should promote that message about subcommand help to someplace more prominent in the overall help message.
I generally print something like this myself:
usage: griffe [GLOBAL_OPTS...] COMMAND [COMMAND_OPTS...]
Signatures for entire Python programs.
Global options:
-h, --help Show this help message and exit. Commands also accept the -h/--help option.
Commands:
COMMAND
dump Load package-signatures and dump them as JSON.
check Check for API breakages or possible improvements.
Not sure if it's more visible ๐คท
Cool - thanks for the suggestion!