QuantConnect/lean-cli

"docker live stop" command failing due to unescaped '$' character

Closed this issue · 1 comments

In my local algorithm deployments I frequently experienced an issue where the command "docker live stop" did not stop my live deployments and resulting in errors like this:

Context information:
  Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
  OS: Linux-6.2.0-33-generic-x86_64-with-glibc2.35
  Lean CLI version: 1.0.169
live.send_command(): stop - sending command. 
live.get_result(): stop -  waiting for results...
Traceback (most recent call last):
  File "/home/constantine/.local/lib/python3.10/site-packages/lean/main.py", line 94, in main
    lean.main(standalone_mode=False)
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/constantine/.local/lib/python3.10/site-packages/lean/click.py", line 159, in invoke
    result = super().invoke(ctx)
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/constantine/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/constantine/.local/lib/python3.10/site-packages/lean/commands/live/stop.py", line 37, in stop
    get_result(command_id, docker_container_name)
  File "/home/constantine/.local/lib/python3.10/site-packages/lean/commands/live/live.py", line 88, in get_result
    raise Exception((f"live.get_result(): {stack()[1].function} - "
Exception: live.get_result(): stop - Failed: to execute the command successfully. Failed to read result from docker file
result-command-6b31a490ac444ef88713ffbb6d082ee0.json within 30 seconds
Error: live.get_result(): stop - Failed: to execute the command successfully. Failed to read result from docker file
result-command-6b31a490ac444ef88713ffbb6d082ee0.json within 30 seconds

I have determined that in the file lean/commands/live/stop.py the issue lies upon the following variable "data":

    data = {
        "$type": "QuantConnect.Commands.AlgorithmStatusCommand, QuantConnect.Common",
        "Id": command_id,
        "Status": "Stopped"
    }

where the "$" character in the json member "$type" is not being escaped when it is passed to the docker exec command in function write_to_file() in lean-cli/lean/components/docker/docker_manager.py and thus fails the write the stop command to the docker container.
I found a workaround by adding a line after line 450 in the write_to_file() function of docker_manager.py to the following:

        data = data.replace('"','\\"')
        data = data.replace('$','\\$')
        command = f'docker exec {docker_container_name} bash -c "echo \'{data}\' > {docker_file.as_posix()}"'

And after adding that line the error no longer occurs

Hey @Grenite!
We believe this was due to a delay on lean engine shutting down, we've increased the command timeout because it was tight and improve the error log. If it happens again please run docker logs {container name} so we can look closer at what is lean doing, which is causing the delay. To get the docker container name you can run docker ps. Release in lean cli version 1.0.171
Thank you, closing the issue for now 👍