Conda installed pre-commit causes problems with graphical git front-ends
Closed this issue · 16 comments
Issue:
Pre-commit stores the python executable used for pre-commit install in the generated git hooks. However, a python installed by conda can only be reliably executed when the respective environment is activated (Especially when using Windows). Thus, when using pre-commit installed by this conda-forge package, the respective environment needs to be activated whenever git commit is called. However, when using a git front-end from an IDE that supports the choice between different conda installed python environments, git will not be called with the correct environment activated.
Pre-commit hook:
$ cat .git/hooks/pre-commit
#!/usr/bin/env python
# File generated by pre-commit: https://pre-commit.com
...
# start templated
INSTALL_PYTHON = 'D:\\Programs\\Miniconda3\\envs\\pre-commit\\python.exe'
...Resulting problem reproduced in Windows cmd (activate environment pre-commit; pre-commit install; git commit -> OK; deactivate environment; git commit -> FAIL):
D:\my\code\try_pre_commit>D:\Programs\Miniconda3\condabin\activate pre-commit
D:\my\code\try_pre_commit>conda.bat activate pre-commit
(pre-commit) D:\my\code\try_pre_commit>pre-commit install
pre-commit installed at .git\hooks\pre-commit
(pre-commit) D:\my\code\try_pre_commit>git commit
black-conda..........................................(no files to check)Skipped
On branch master
nothing to commit, working tree clean
(pre-commit) D:\my\code\try_pre_commit>D:\Programs\Miniconda3\condabin\deactivate
DeprecationWarning: 'deactivate' is deprecated. Use 'conda deactivate'.
(pre-commit) D:\my\code\try_pre_commit>conda.bat deactivate
D:\my\code\try_pre_commit>git commit
Traceback (most recent call last):
File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\__main__.py", line 1, in <module>
from pre_commit.main import main
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\main.py", line 13, in <module>
from pre_commit.commands.autoupdate import autoupdate
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\commands\autoupdate.py", line 17, in <module>
from pre_commit.clientlib import InvalidManifestError
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\clientlib.py", line 16, in <module>
from pre_commit.error_handler import FatalError
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\error_handler.py", line 10, in <module>
from pre_commit.store import Store
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\store.py", line 4, in <module>
import sqlite3
File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.
Environment (
conda list):
# packages in environment at D:\Programs\Miniconda3\envs\pre-commit:
#
# Name Version Build Channel
aspy.yaml 1.3.0 py_0 conda-forge
ca-certificates 2019.11.28 hecc5488_0 conda-forge
certifi 2019.11.28 py38_0 conda-forge
cfgv 2.0.1 py_0 conda-forge
editdistance 0.5.3 py38h6538335_0 conda-forge
identify 1.4.11 py_0 conda-forge
nodeenv 1.3.4 py_0 conda-forge
openssl 1.1.1d hfa6e2cd_0 conda-forge
pip 20.0.2 py38_0 conda-forge
pre-commit 2.0.0 py38_0 conda-forge
python 3.8.1 he1f5543_2 conda-forge
pyyaml 5.3 py38hfa6e2cd_0 conda-forge
setuptools 45.1.0 py38_0 conda-forge
six 1.14.0 py38_0 conda-forge
sqlite 3.30.1 hfa6e2cd_0 conda-forge
toml 0.10.0 py_0 conda-forge
vc 14.1 h0510ff6_4
virtualenv 16.7.5 py_0 conda-forge
vs2015_runtime 14.16.27012 hf0eaf9b_1
wheel 0.34.1 py38_0 conda-forge
wincertstore 0.2 py38_1003 conda-forge
yaml 0.2.2 hfa6e2cd_1 conda-forge
Details about
conda and system ( conda info ):
active environment : pre-commit
active env location : D:\Programs\Miniconda3\envs\pre-commit
shell level : 1
user config file : C:\Users\me\.condarc
populated config files :
conda version : 4.7.12
conda-build version : not installed
python version : 3.7.4.final.0
virtual packages :
base environment : D:\Programs\Miniconda3 (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
package cache : D:\Programs\Miniconda3\pkgs
C:\Users\me\.conda\pkgs
C:\Users\me\AppData\Local\conda\conda\pkgs
envs directories : D:\Programs\Miniconda3\envs
C:\Users\me\.conda\envs
C:\Users\me\AppData\Local\conda\conda\envs
platform : win-64
user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.4 Windows/10 Windows/10.0.18362
administrator : False
netrc file : None
offline mode : False
I tried to propose a pull request to the maintainer of pre-commit, but he considers it a bug of conda that conda installed python executables do not work out of the box:
pre-commit/pre-commit#1329
The following pull request records the conda environment used for calling pre-commit install when using a specific option and runs pre-commit hooks using conda run:
pre-commit/pre-commit#1324
It would be possible to propose a much simpler change if the conda environment is not recorded on pre-commit install, but simply taken from an environment variable that must be set when calling git commit.
I tried to propose a pull request to the maintainer of pre-commit, but he considers it a bug of conda that conda installed python executables do not work out of the box:
pre-commit/pre-commit#1329
And I agree, this is mostly a conda issue, where environments need to be activated in order to be used properly. With Python virtual environments, this is never really needed, you can call python from the virtual environment and things just work.
Also, this is not something specific to pre-commit, any tool which calls a python installed by conda might suffer this.
To work around this, we use https://github.com/gqmelo/exec-wrappers (with https://github.com/conda-forge/conda-wrappers-feedstock), which basically solves the issue universally: just install the conda-wrapper package into your environment, and you will have .bat files generated for you inside your environment that activate the environment first before calling the wrapped executable. Then when you install pre-commit using the python.bat executable, it will record the full path to python.bat and things will just work.
We also use the same python.bat to register the interpreter in IDEs (such as PyCharm and LiClipse), and it works great.
cc @gqmelo
In general always expect anything installed using conda to work only in an activated environment.
There is a lot happening to make the packages relocatable and many of them rely on environment variables being set (and this happens during activation).
This is the reason why I created the exec-wrappers and conda-wrappers mentioned by nicoddemus. It creates wrappers that mimic the conda activation before running the real executable.
In the case of pre-commit, I think you have some options:
Use git from conda-forge + wrappers
Install git from conda-forge, create the wrappers and use the git wrapper instead of the git installed in your system:
conda install git
conda install conda-wrappersYou should find a git wrapper in D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\git.bat. Running it will ensure that git runs in the activated environment.
Create wrappers for your current git installation
If you don't want to install git from conda-forge, you can wrap your current git with something like:
create-wrappers -t conda -b C:\Program Files\Git\bin -d PATH_WHERE_TO_CREATE_WRAPPERS --conda-env-dir D:\Programs\Miniconda3\envs\pre-commit
Then use the git wrappers from PATH_WHERE_TO_CREATE_WRAPPERS
Change generated pre-commit files to use python.bat wrapper
Yet another option would be to change the python executable in the pre commit files to be D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\python.bat
This option could be easier if there were an option on pre-commit to override the python executable to be saved in the files. Currently it uses sys.executable. Maybe providing a command line option to override it is something the author is willing to accept.
Thank you so much for the great hints and the well structured explanation!
The conda-wrappers package is actually pretty close to what I would see as an alternative to the conda activate system. The problem of conda activate is that it needs to hack the calling shell. I would put a pre-computed conda activate (mostly just environment variable settings) in some place of the environment on every conda install action and then have wrapper executables for all entry points that apply those before executing their real body. If they are binary injected versions of the real executables or of they include a hack changing sys.executable, conda activation would not be needed on programmatic calls.
A quick try of git.bat did not work out of the box.
D:\Programs\cygwin\home\me\code\try_pre_commit>D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\git.bat commit
Traceback (most recent call last):
File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\__main__.py", line 1, in <module>
from pre_commit.main import main
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\main.py", line 13, in <module>
from pre_commit.commands.autoupdate import autoupdate
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\commands\autoupdate.py", line 17, in <module>
from pre_commit.clientlib import InvalidManifestError
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\clientlib.py", line 16, in <module>
from pre_commit.error_handler import FatalError
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\error_handler.py", line 10, in <module>
from pre_commit.store import Store
File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\store.py", line 4, in <module>
import sqlite3
File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.
I am missing the activate.d directory mentioned in runin.bat:
$ find /cygdrive/d/Programs/Miniconda3/envs/pre-commit -iname "*activate*"
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/common/activate
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/common/Activate.ps1
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/nt/activate.bat
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/nt/deactivate.bat
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/posix/activate.csh
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/posix/activate.fish
@gqmelo, did you consider using conda run? A colleague of mine tries to get a pull request upstream which would add an option to conda run that makes the output interactive:
conda/conda#9646
Calling Lib/venv/scripts/nt/activate.bat in a modified runin.bat also does not work. conda run -n pre-commit git commit does work.
Not sure I have a similar issue (the same error message though) but when I call the precommit executable directly (full path) without activating the conda environment on Windows, it's also causing failing builds. If I get this right, stand-alone executables should not be installed with conda because they are not ultimately standalone and depend on activated envs. This sounds like bad news to me. Hard for me to investigate now, but it seems it works when there is only one conda environment because that's probably activated by default. When installing into another env, the sqlite error occurs.
@QuantCo-mr-T I solved my problem with conda run -n env-name path/to/pre-commit/exec arg1 arg2, so I don't need to activate the environment in a separate command. This does not apply to the original issue though I think.
Then use the git wrappers from
PATH_WHERE_TO_CREATE_WRAPPERSChange generated pre-commit files to use
python.batwrapperYet another option would be to change the python executable in the pre commit files to be
D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\python.batThis option could be easier if there were an option on
pre-committo override the python executable to be saved in the files. Currently it usessys.executable. Maybe providing a command line option to override it is something the author is willing to accept.
@gqmelo
Which file in the pre-commit package do I change to point to use the python.bat file?
Change generated pre-commit files to use
python.batwrapperYet another option would be to change the python executable in the pre commit files to be
D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\python.bat
In case anyone else comes across this issue, there appears to be an even simpler fix:
If using windows, activate your conda env in the terminal and then activate your editor from the terminal as well:
eg, . code for vs code or PyCharm\bin\pycharm.bat for PyCharm - suggested by @lancelote https://stackoverflow.com/questions/63663562/use-of-pre-commit-with-pycharm#comment112601701_63663562
This should be resolved by using the latest Python 3.9 and 3.10 Python builds. No need for workarounds anymore.
This should be resolved by using the latest Python 3.9 and 3.10 Python builds. No need for workarounds anymore.
I'm facing this issue using vscode, Conda with Python 3.10.6 and pre-commit 2.20.0
@master117 Please post a traceback or other supporting information.
@master117 Please post a traceback or other supporting information.
[2022-10-19T15:23:40.685Z] Traceback (most recent call last):
File "C:\Users\<user>\Miniconda3\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\<user>\Miniconda3\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\<user>\Miniconda3\lib\site-packages\pre_commit\__main__.py", line 3, in <module>
from pre_commit.main import main
File "C:\Users\<user>\Miniconda3\lib\site-packages\pre_commit\main.py", line 12, in <module>
from pre_commit.commands.autoupdate import autoupdate
File "C:\Users\<user>\Miniconda3\lib\site-packages\pre_commit\commands\autoupdate.py", line 18, in <module>
from pre_commit.store import Store
File "C:\Users\<user>\Miniconda3\lib\site-packages\pre_commit\store.py", line 6, in <module>
import sqlite3
File "C:\Users\<user>\Miniconda3\lib\sqlite3\__init__.py", line 57, in <module>
from sqlite3.dbapi2 import *
File "C:\Users\<user>\Miniconda3\lib\sqlite3\dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.
If I'm stating vscode from a prompt with activate conda environemnt it works.
Can you please also post a conda list -n base here?
Can you please also post a
conda list -n basehere?
Of course.
# packages in environment at C:\Users\<user>\Miniconda3:
#
# Name Version Build Channel
brotlipy 0.7.0 py39h2bbff1b_1003
ca-certificates 2022.9.24 h5b45459_0 conda-forge
certifi 2022.9.24 pyhd8ed1ab_0 conda-forge
cffi 1.15.0 py39h2bbff1b_1
cfgv 3.3.1 pyhd8ed1ab_0 conda-forge
charset-normalizer 2.0.4 pyhd3eb1b0_0
colorama 0.4.4 pyhd3eb1b0_0
conda 22.9.0 py39hcbf5309_1 conda-forge
conda-content-trust 0.1.1 pyhd3eb1b0_0
conda-package-handling 1.8.1 py39h8cc25b3_0
console_shortcut 0.1.1 4
cryptography 36.0.0 py39h21b164f_0
distlib 0.3.6 pypi_0 pypi
filelock 3.8.0 pyhd8ed1ab_0 conda-forge
identify 2.5.6 pyhd8ed1ab_0 conda-forge
idna 3.3 pyhd3eb1b0_0
menuinst 1.4.18 py39h59b6b97_0
nodeenv 1.7.0 pyhd8ed1ab_0 conda-forge
openssl 1.1.1q h8ffe710_0 conda-forge
pip 21.2.4 py39haa95532_0
platformdirs 2.5.2 pyhd8ed1ab_1 conda-forge
powershell_shortcut 0.0.1 3
pre-commit 2.20.0 py39hcbf5309_0 conda-forge
pycosat 0.6.3 py39h2bbff1b_0
pycparser 2.21 pyhd3eb1b0_0
pyopenssl 22.0.0 pyhd3eb1b0_0
pysocks 1.7.1 py39haa95532_0
python 3.9.12 h6244533_0
python_abi 3.9 2_cp39 conda-forge
pywin32 302 py39h2bbff1b_2
pyyaml 6.0 py39hb82d6ee_4 conda-forge
requests 2.27.1 pyhd3eb1b0_0
ruamel_yaml 0.15.100 py39h2bbff1b_0
setuptools 61.2.0 py39haa95532_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.38.2 h2bbff1b_0
toml 0.10.2 pyhd8ed1ab_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tqdm 4.63.0 pyhd3eb1b0_0
tzdata 2022a hda174b7_0
ukkonen 1.0.1 py39h2e07f2f_2 conda-forge
urllib3 1.26.8 pyhd3eb1b0_0
vc 14.2 h21ff451_1
virtualenv 20.16.5 py39hcbf5309_0 conda-forge
vs2015_runtime 14.27.29016 h5e58377_2
wheel 0.37.1 pyhd3eb1b0_0
win_inet_pton 1.1.0 py39haa95532_0
wincertstore 0.2 py39haa95532_2
yaml 0.2.5 he774522_0
This was run inside a powershell at the root folder of the project, using the base or no environment.
This issue here is that the Python is your base environment probably doesn't have the necessary bugfix. As reported by conda list is from the main channel and not from conda-forge. Can you try to install conda-forge::python=3.9.13 in your base environment and recheck whether pre-commit works?