neuro-inc/neuro-cli

ERROR: can't concat WebSocketError to bytes

Closed this issue · 3 comments

andriihomiak commented

Description:

Sometimes neuro fails with the following message:

ERROR: can't concat WebSocketError to bytes

After which the logs are not saved

Example:

Seems to happen randomly when running any job via neuro run.
Example logs with traceback:

Traceback (most recent call last):
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/main.py", line 557, in main
    cli.main(
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/utils.py", line 332, in invoke
    super().invoke(ctx)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/utils.py", line 298, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/utils.py", line 110, in wrapper
    return root.run(
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/root.py", line 148, in run
    return self._runner.run(main)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/asyncio_utils.py", line 53, in run
    return self._loop.run_until_complete(main_task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/utils.py", line 82, in _run_async_function
    return await func(root, *args, **kwargs)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/job.py", line 229, in logs
    await process_logs(
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/ael.py", line 108, in process_logs
    txt = decoder.decode(chunk)
  File "/usr/lib/python3.8/codecs.py", line 321, in decode
    data = self.buffer + input
TypeError: can't concat WebSocketError to bytes

Expected behavior:

No error is shown and logs are accessible

Additional context:

Neuro Platform Client 22.6.0
Linux pop-os 5.17.9-051709-generic #202205230806-Ubuntu SMP PREEMPT Mon May 23 08:22:33 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
serhiy-storchaka commented

It was partially fixed in #2746 (at least you should now get a different error in this case). Wait for the next release today.

  • 👍1
andriihomiak commented

Yes, still get the error, but with a different - more informative - message. Repurposed into #2788