bskinn/sphobjinv

Reading compressed inventory from stdin results in decoding error

Closed this issue · 12 comments

pawamoy commented

Brief description

Trying to display an inventory in the terminal, by curling it and piping it to sphobjinv convert:

curl https://pip.pypa.io/en/latest/objects.inv -o- | sphobjinv convert plain - -

Expected behavior
sphobjinv should read stdin, try to decode it, but catch exceptions and consider it compressed.

Actual behavior

sphobjinv doesn't catch the UnicodeDecodeError.

To reproduce

% curl https://pip.pypa.io/en/latest/objects.inv -o- | sphobjinv convert plain - -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 
100  9485  100  9485    0     0  14280      0 --:--:-- --:--:-- --:--:-- 14306
Traceback (most recent call last):
  File "/home/pawamoy/.local/bin/sphobjinv", line 8, in <module>
    sys.exit(main())
  File "/home/pawamoy/.local/pipx/venvs/sphobjinv/lib/python3.10/site-packages/sphobjinv/cli/core.py", line 88, in main
    inv = inv_stdin(params)
  File "/home/pawamoy/.local/pipx/venvs/sphobjinv/lib/python3.10/site-packages/sphobjinv/cli/load.py", line 245, in inv_stdin
    data = sys.stdin.read()
  File "/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.10.8/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 119: invalid continuation byte

System information

  • Device: Computer?
  • OS: ArchLabs

Python environment

Python
3.11

Libraries

% pipx runpip sphobjinv freeze 
attrs==22.2.0
certifi==2022.12.7
jsonschema==4.17.3
pyrsistent==0.19.3
sphobjinv==2.3.1

Additional context

Passing a temporary file descriptor to the command does not work either:

% sphobjinv convert plain <(curl -s https://pip.pypa.io/en/latest/objects.inv -o-) -


Error while parsing input file path:
FileNotFoundError: Indicated path is not a valid file
bskinn commented

I looked into this a bit when I implemented the stdin stream capability, and something about running a compressed inventory through a pipe appears to mangle it---no matter what I tried, I couldn't get it to recognize correctly.

I'm guessing it has something to do with character encoding, as AFAIK the *nix piping system expects to be passing plaintext, not bytestreams. There could be an env variable out there you can set to change the encoding behavior (a "raw" pipe mode?), in which case it might be workable.

I didn't want to put the work in to figure that out until I was sure it wasn't YAGNI. Which--now, it's not. 😬

If you have any thoughts about things to try, please let me know!

bskinn commented

Although this makes me think I'm wrong about this, in which case I'm not sure what went wrong.

Maybe at the *nix/Python interface....

pawamoy commented

Oh, I see, thanks for the prompt reply!

It seems hard to get right indeed, so I'll just tell you that you can close this, because the workaround is so easy: just actually download the inv file locally, then run sphobjinv.

I think you also have tickets open for accepting URLs, that would solve the issue as well, and provide an even better UX (no pipe needed, more cross-platform, etc) 👍

bskinn commented

Hm. The feature is already there for pulling inventories directly from URLs.

Did you try:

% sphobjinv convert plain -u https://pip.pypa.io/en/latest/objects.inv -

?

pawamoy commented

Ah, my bad, I glanced at the issues and the usage (-h) but didn't understand this was possible. Well that's exactly what I needed haha, thanks!

  • 👍1
bskinn commented

Aha, I need to add to the base help message (sphobjinv -h) to indicate that there's additional help for each subcommand (sphobjinv convert -h and sphobjinv suggest -h).

pawamoy commented

It's already the case 😄

% sphobjinv -h
usage: sphobjinv [-h] [-v] {convert,suggest} ...

Format conversion for and introspection of intersphinx 'objects.inv' files.

options:
  -h, --help         show this help message and exit
  -v, --version      Print package version & other info

Subcommands:
  {convert,suggest}  Execution mode. Type 'sphobjinv [mode] -h' for more information on available options. Mode names can be abbreviated to their first two letters.
    convert (co)     Convert intersphinx inventory to zlib-compressed, plaintext, or JSON formats.
    suggest (su)     Fuzzy-search intersphinx inventory for desired object(s).

I had just missed the -u option in the help for the convert subcommand 🙂

bskinn commented

Oh hey! So it is. I scanned my own help message too quickly... 😅

bskinn commented

Though -- since you missed it, probably means I should promote that message about subcommand help to someplace more prominent in the overall help message.

pawamoy commented

I generally print something like this myself:

usage: griffe [GLOBAL_OPTS...] COMMAND [COMMAND_OPTS...]

Signatures for entire Python programs.

Global options:
  -h, --help  Show this help message and exit. Commands also accept the -h/--help option.

Commands:
  COMMAND
    dump      Load package-signatures and dump them as JSON.
    check     Check for API breakages or possible improvements.

Not sure if it's more visible 🤷

  • 👍1
bskinn commented

Cool - thanks for the suggestion!

bskinn commented

Closing since the original issue was resolved, and the help message change spun into #282.