DVC fails to read path to python.exe, if forward slashes "/ " used in windows
HarryKalantzopoulos opened this issue · 6 comments
Bug Report
dvc repro fails to read path/to/python.exe, in windows, because of slashes "/". There is no problem with path/to/my_script.py
Issue name
repro: failed to reproduce 'test': "'.venv' is not recognized as an internal or external command. ERROR: failed to run: .venv/Scripts/python.exe script/test.py"
Description
I try to use another python executable originated in my virtual environment. I prefer to use forward slashes, as sometimes I want to test them also in WSL. Even if I can run the command line with slashes, dvc repro requires forward slashes.
Reproduce
git init -q
dvc init -q
python -m venv .venv
dvc stage add -n test .venv/Scripts/python.exe script/test.py
dvc repro # won't work
dvc stage add -f -n test .venv\Scripts\python.exe script/test.py
dvc repro # will work
# This will also work
.venv/Scripts/python.exe script/test.py
Expected
I expect from a Linux OS to not run with backslashes, but Windows doesn't have any issue either with backslashes or forward slashes in the path.
Environment information
dvc == 3.26.0
Hi, could you please provide a more verbose output? Also, please try with latest release.
Hello @skshetry .
Alright I tried DVC v.3.48.1, but the error persists. Here what I got from dvc repro -v
Verbose output
2024-03-02 11:27:56,081 DEBUG: v3.48.1 (conda), CPython 3.12.2 on Windows-11-10.0.22624-SP0
2024-03-02 11:27:56,081 DEBUG: command: C:\Users\ham_p\miniconda3\envs\testing\Scripts\dvc repro -v
2024-03-02 11:27:56,826 DEBUG: Lockfile 'dvc.lock' needs to be updated.
2024-03-02 11:27:57,527 DEBUG: stage: 'test' changed.
Running stage 'test':
> .venv/Scripts/python.exe script/test.py
'.venv' is not recognized as an internal or external command,
operable program or batch file.
2024-03-02 11:27:57,602 ERROR: failed to reproduce 'test': failed to run: .venv/Scripts/python.exe script/test.py, exited with 1
Traceback (most recent call last):
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\repo\reproduce.py", line 187, in _reproduce
ret = repro_fn(stage, upstream=upstream, force=force_stage, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\repo\reproduce.py", line 117, in _reproduce_stage
ret = stage.reproduce(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\funcy\decorators.py", line 47, in wrapper
return deco(call, *dargs, **dkwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\decorators.py", line 44, in rwlocked
return call()
^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\funcy\decorators.py", line 68, in __call__
return self._func(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\__init__.py", line 437, in reproduce
self.run(**kwargs)
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\funcy\decorators.py", line 47, in wrapper
return deco(call, *dargs, **dkwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\decorators.py", line 44, in rwlocked
return call()
^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\funcy\decorators.py", line 68, in __call__
return self._func(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\__init__.py", line 605, in run
self._run_stage(dry, force, **kwargs)
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\funcy\decorators.py", line 47, in wrapper
return deco(call, *dargs, **dkwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\decorators.py", line 44, in rwlocked
return call()
^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\funcy\decorators.py", line 68, in __call__
return self._func(*self._args, **self._kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\__init__.py", line 636, in _run_stage
return run_stage(self, dry, force, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\run.py", line 138, in run_stage
run(stage, dry=dry, run_env=run_env)
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\decorators.py", line 53, in wrapper
ret = f(stage, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\run.py", line 113, in cmd_run
_run(executable, cmd, **kwargs)
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\stage\run.py", line 93, in _run
raise StageCmdFailedError(cmd, p.returncode)
dvc.stage.exceptions.StageCmdFailedError: failed to run: .venv/Scripts/python.exe script/test.py, exited with 1
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\cli\__init__.py", line 211, in main
ret = cmd.do_run()
^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\cli\command.py", line 27, in do_run
return self.run()
^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\commands\repro.py", line 11, in run
stages = self.repo.reproduce(**self._common_kwargs, **self._repro_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\repo\__init__.py", line 58, in wrapper
return f(repo, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\repo\scm_context.py", line 143, in run
return method(repo, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\repo\reproduce.py", line 248, in reproduce
return _reproduce(steps, graph=graph, on_error=on_error or "fail", **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\repo\reproduce.py", line 191, in _reproduce
_raise_error(exc, stage)
File "C:\Users\ham_p\miniconda3\envs\testing\Lib\site-packages\dvc\repo\reproduce.py", line 155, in _raise_error
raise ReproductionError(f"failed to reproduce{segment} {names}") from exc
dvc.exceptions.ReproductionError: failed to reproduce 'test'
2024-03-02 11:27:57,621 DEBUG: Analytics is enabled.
2024-03-02 11:27:57,897 DEBUG: Trying to spawn ['daemon', 'analytics', 'C:\\Users\\ham_p\\AppData\\Local\\Temp\\tmp39nmppdp', '-v']
2024-03-02 11:27:57,903 DEBUG: Spawned ['daemon', 'analytics', 'C:\\Users\\ham_p\\AppData\\Local\\Temp\\tmp39nmppdp', '-v'] with pid 30796Hi, which shell are you using? It might be working due to git-bash.
Does that command work in CMD? Windows does not support forward slashes in every place and is not very consistent.
As you can see above, dvc is passing commands directly to the subprocess.run.
I suppose you are right. I use powershell, but subprocess should be using cmd (my COMSPEC is cmd). However, it does not face any problem with path/to/python_file. I suppose you are correcting the path to the file, but not the path to the executable?
e.g. if cmd: .venv\Scripts\python.exe script/test.py. dvc repro will work.
DVC is not changing anything. Windows supports forward slashes for the files. My guess is that it does not always work for executables.
You are right. Thank you for your time!