python-lsp-ruff is using the globally installed ruff instead of the one of the virtual environment
nicoe opened this issue · 20 comments
My setup is the following
NVIM v0.9.1
pylsp: 1.7.3
ruff: 0.0.267
The issue
Linting a file where I know there are issues does not work.
I've got the following traceback in the log:
2023-05-16 11:18:42,852 CEST - INFO - pylsp.config.config - Updated settings to {'plugins': {'mccabe': {'enabled': False}, 'ruff': {'extendSelect': ['I'], 'enabled': True}}, 'configurationSources': ['flake8']}
2023-05-16 11:18:42,853 CEST - INFO - pylsp.config.config - Disabled plugins: [<module 'pyls_isort.plugin' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pyls_isort/plugin.py'>, <module 'pylsp.plugins.flake8_lint' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/flake8_lint.py'>, <module 'pylsp.plugins.mccabe_lint' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/mccabe_lint.py'>, <module 'pylsp.plugins.pycodestyle_lint' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/pycodestyle_lint.py'>, <module 'pylsp.plugins.pydocstyle_lint' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/pydocstyle_lint.py'>, <module 'pylsp.plugins.pyflakes_lint' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/pyflakes_lint.py'>, <module 'pylsp.plugins.pylint_lint' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/pylint_lint.py'>, <module 'pylsp.plugins.rope_autoimport' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/rope_autoimport.py'>, <module 'pylsp.plugins.rope_completion' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/rope_completion.py'>, <module 'pylsp.plugins.rope_rename' from '/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/plugins/rope_rename.py'>]
2023-05-16 11:18:43,386 CEST - WARNING - pylsp.config.config - Failed to load hook pylsp_lint: While structuring typing.List[pylsp_ruff.ruff.Check] (1 sub-exception)
+ Exception Group Traceback (most recent call last):
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp/config/config.py", line 33, in _hookexec
| return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 327, in traced_hookexec
| return outcome.get_result()
| ^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
| raise ex[1].with_traceback(ex[2])
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pluggy/_result.py", line 33, in from_call
| result = func()
| ^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 324, in <lambda>
| lambda: oldcall(hook_name, hook_impls, kwargs, firstresult)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
| return outcome.get_result()
| ^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
| raise ex[1].with_traceback(ex[2])
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
| res = hook_impl.function(*args)
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp_ruff/plugin.py", line 125, in pylsp_lint
| checks = run_ruff_check(workspace, document)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/pylsp_ruff/plugin.py", line 354, in run_ruff_check
| return converter.structure(result, List[RuffCheck])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/cattrs/converters.py", line 309, in structure
| return self._structure_func.dispatch(cl)(obj, cl)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/cattrs/converters.py", line 510, in _structure_list
| raise IterableValidationError(
| cattrs.errors.IterableValidationError: While structuring typing.List[pylsp_ruff.ruff.Check] (1 sub-exception)
+-+---------------- 1 ----------------
| Exception Group Traceback (most recent call last):
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/cattrs/converters.py", line 502, in _structure_list
| res.append(handler(e, elem_type))
| ^^^^^^^^^^^^^^^^^^^^^
| File "<cattrs generated structure pylsp_ruff.ruff.Check>", line 35, in structure_Check
| if errors: raise __c_cve('While structuring ' + 'Check', errors, __cl)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| cattrs.errors.ClassValidationError: While structuring Check (1 sub-exception)
| Structuring typing.List[pylsp_ruff.ruff.Check] @ index 0
+-+---------------- 1 ----------------
| Exception Group Traceback (most recent call last):
| File "<cattrs generated structure pylsp_ruff.ruff.Check>", line 31, in structure_Check
| res['fix'] = __c_structure_fix(o['fix'], __c_type_fix)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/lib/python3.11/site-packages/cattrs/converters.py", line 574, in _structure_optional
| return self._structure_func.dispatch(other)(obj, other)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "<cattrs generated structure pylsp_ruff.ruff.Fix>", line 14, in structure_Fix
| if errors: raise __c_cve('While structuring ' + 'Fix', errors, __cl)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| cattrs.errors.ClassValidationError: While structuring Fix (1 sub-exception)
| Structuring class Check @ attribute fix
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "<cattrs generated structure pylsp_ruff.ruff.Fix>", line 5, in structure_Fix
| res['edits'] = __c_structure_edits(o['edits'], __c_type_edits)
| ~^^^^^^^^^
| KeyError: 'edits'
| Structuring class Fix @ attribute edits
+------------------------------------
I tried using an older version of ruff but it didn't work (I haven't tested with older version of pylsp because uninstalling python-lsp-ruff did bring the linting back).
Can you also post the versions of python-lsp-ruff and cattrs?
Yes of course 🙂 :
python-lsp-ruff: 1.4.0
cattrs: 22.2.0
And BTW I have to uninstall python-lsp-ruff, disabling it do not enable flake8 back.
It looks like #11 but it's not the same as there is no linting and no error in the log (ruff is not called), but flake8 is not called either. But this is probably another issue.
Hmm, the issue indicates that something in the API between ruff, pylsp_ruff and possibly lsprotocol changed. What version of lsprotocol do you use?
Hi there,
I have the same problem with emacs and lsp-mode.
Here are all the related packages I have installed (on archlinux yay -Qs lsp python):
python-lsp-black 1.2.1-2
python-lsp-jsonrpc 1.0.0-4
python-lsp-ruff 1.4.0-1
python-lsp-server 1.7.2-3
python-lsprotocol 2023.0.0a1-2
Doom Emacs and lsp-mode does not support the ruff plugin explicitly, so the following snippet is also required:
(after! lsp-mode
(defcustom-lsp lsp-pylsp-plugin-ruff-enabled t
"enable ruff linting."
:group 'lsp-pylsp
:type 'boolean
:lsp-path "pylsp.plugins.ruff.enabled")
)Hmm, the issue indicates that something in the API between ruff, pylsp_ruff and possibly lsprotocol changed. What version of lsprotocol do you use?
Just like @paquiteau : lsprotocol==2023.0.0a1
A potential workaround is to run ruff-lsp alongside pylsp (LSP-mode does all the heavy lifting for this in Emacs). I don't see (yet) any performance impact of running two LSP clients side-by-side.
I still have trouble reproducing this. Can you provide a minimal working example?
@nicoe I am maintaining an extension called coc-pylsp in the coc.nvim plugin for Vim/Neovim, and it seems to be working fine with python-lsp-ruff (v1.4.0)
Based on the error logs, it appears to be related to a feature recently added in the latest version of python-lsp-ruff. If you try downgrading the version of python-lsp-ruff, it might work for now.
Okay, so the only way to check what is going on is to manually check what the list that cattrs tries to structure. Can you put a log.warning just before the structure line in pylsp_ruff/plugin.py l.354, e.g.
log.warn(f"Erroneous list: {result}")? This way you will find it easier in the logs. (Make sure the edited version of the package is used)
Also, I wonder if this comes from a specific Ruff error? Can you also send your pylsp/ruff config?
Okay, so the only way to check what is going on is to manually check what the list that
cattrstries to structure. Can you put alog.warningjust before the structure line inpylsp_ruff/plugin.pyl.354, e.g.log.warn(f"Erroneous list: {result}")? This way you will find it easier in the logs. (Make sure the edited version of the package is used)
Sorry for the delay, I was travelling for work and had few time to come back to this.
So, I tested again with your suggestion and although nothing else has changed, I can not reproduced the error anymore also. I don't know why though …
So if @paquiteau does not have the issue either I think it can be safely closed.
OK I spoke too soon because I was checking with a small file I made for this.
Using a real file here is the result of the log
2023-05-26 17:29:54,000 CEST - WARNING - pylsp_ruff.plugin - Erroneous list: [{'code': 'I001', 'message': 'Import block is un-sorted or un-formatted', 'fix': {'content': 'import datetime as dt\nfrom collections import defaultdict, namedtuple\nfrom decimal import Decimal\nfrom itertools import chain, combinations, groupby\n\nfrom sql import Literal, Null\nfrom sql.aggregate import Sum\nfrom sql.conditionals import Case, Coalesce\nfrom sql.functions import CharLength, Round\nfrom sql.operators import Exists\nfrom trytond import backend\nfrom trytond.config import config\nfrom trytond.i18n import gettext\nfrom trytond.model import (\n DeactivableMixin,\n Index,\n ModelSQL,\n ModelView,\n Unique,\n Workflow,\n dualmethod,\n fields,\n sequence_ordered,\n)\nfrom trytond.model.exceptions import AccessError\nfrom trytond.modules.account.tax import TaxableMixin\nfrom trytond.modules.currency.fields import Monetary\nfrom trytond.modules.product import price_digits\nfrom trytond.pool import Pool\nfrom trytond.pyson import Bool, Eval, If\nfrom trytond.report import Report\nfrom trytond.rpc import RPC\nfrom trytond.tools import firstline, grouped_slice, reduce_ids\nfrom trytond.transaction import Transaction\nfrom trytond.wizard import Button, StateAction, StateTransition, StateView, Wizard\n\nfrom .exceptions import (\n InvoiceFutureWarning,\n InvoiceNumberError,\n InvoicePaymentTermDateWarning,\n InvoiceSimilarWarning,\n InvoiceTaxValidationError,\n InvoiceValidationError,\n PayInvoiceError,\n)\n\n', 'message': 'Organize imports', 'location': {'row': 3, 'column': 0}, 'end_location': {'row': 38, 'column': 0}}, 'location': {'row': 3, 'column': 1}, 'end_location': {'row': 38, 'column': 1}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 3}, {'code': 'F821', 'message': 'Undefined name `ls`', 'fix': None, 'location': {'row': 651, 'column': 21}, 'end_location': {'row': 651, 'column': 23}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 651}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 768, 'column': 26}, 'end_location': {'row': 768, 'column': 27}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 768}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 768, 'column': 56}, 'end_location': {'row': 768, 'column': 57}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 768}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 1085, 'column': 44}, 'end_location': {'row': 1085, 'column': 45}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 1085}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 1227, 'column': 35}, 'end_location': {'row': 1227, 'column': 36}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 1227}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 1286, 'column': 40}, 'end_location': {'row': 1286, 'column': 41}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 1286}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 1393, 'column': 19}, 'end_location': {'row': 1393, 'column': 20}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 1393}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 1503, 'column': 32}, 'end_location': {'row': 1503, 'column': 33}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 1503}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 1504, 'column': 31}, 'end_location': {'row': 1504, 'column': 32}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 1504}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 1884, 'column': 53}, 'end_location': {'row': 1884, 'column': 54}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 1884}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 3337, 'column': 39}, 'end_location': {'row': 3337, 'column': 40}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 3337}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 3339, 'column': 27}, 'end_location': {'row': 3339, 'column': 28}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 3339}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 3419, 'column': 26}, 'end_location': {'row': 3419, 'column': 27}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 3419}, {'code': 'E741', 'message': 'Ambiguous variable name: `l`', 'fix': None, 'location': {'row': 3433, 'column': 26}, 'end_location': {'row': 3433, 'column': 27}, 'filename': '/home/nicoe/projets/tryton/tryton/modules/account_invoice/invoice.py', 'noqa_row': 3433}]
I think that the issue comes from the I001 error. But I do not reproduce it every time (even on the same file but with different invocations of nvim).
During the creation of the new release I realized I had set the tag incorrectly which might have produced your errors.
The new release should be tagged right so please try to update the package and see if this error is still present in the new version.
I still get same error. It occurs without I001.
I dug a little deeper into the log and found that your ruff version is not up-to-date. The format that you get is
{
"code":"I001",
"message":"Import block is un-sorted or un-formatted",
"fix":{
"content":"import datetime as dt\n...",
"message":"Organize imports",
"location":{
"row":3,
"column":0
},
"end_location":{
"row":38,
"column":0
}
}
}This is what I get when using ruff v0.0.267:
"code":"I001",
"message":"Import block is un-sorted or un-formatted",
"fix":{
"applicability":"Unspecified",
"message":"Organize imports",
"edits":[
{
"content":"import numpy as np\nimport scipy\n",
"location":{
"row":1,
"column":1
},
"end_location":{
"row":3,
"column":13
}
}
]
},Please make sure that your version of ruff is up-to-date. @nicoe for you that means first going into the mason environment, e.g. via source /home/nicoe/.local/share/nvim/mason/packages/python-lsp-server/venv/activate and then doing a pip update for ruff (as well as python-lsp-server and python-lsp-ruff).
In the latest version ruff must be at least v.0.0.267.
I dug a little deeper into the log and found that your ruff version is not up-to-date.
Thank you for spending time on this issue.
But I doubt this is the case. Here's a transcript of my shell session (slightly edited to emphasize ruff):
~
❯ cd .local/share/nvim/mason/packages/python-lsp-server
~/.l/s/n/mason/packages/python-lsp-server
❯ ls
mason-schemas venv mason-receipt.json
~/.l/s/n/mason/packages/python-lsp-server
❯ source venv/bin/activate
~/.l/s/n/mason/packages/python-lsp-server (venv):v3.11.2
❯ pip freeze
astroid==2.15.5
attrs==23.1.0
autopep8==2.0.2
cattrs==22.2.0
dill==0.3.6
docstring-to-markdown==0.12
flake8==6.0.0
isort==5.12.0
jedi==0.18.2
lazy-object-proxy==1.9.0
lsprotocol==2023.0.0a1
mccabe==0.7.0
packaging==23.1
parso==0.8.3
platformdirs==3.5.1
pluggy==1.0.0
pycodestyle==2.10.0
pydocstyle==6.3.0
pyflakes==3.0.1
pylint==2.17.4
python-lsp-isort==0.1
python-lsp-jsonrpc==1.0.0
python-lsp-ruff==1.5.0
python-lsp-server==1.7.3
pytoolconfig==1.2.5
rope==1.8.0
ruff==0.0.267
snowballstemmer==2.2.0
tomli==2.0.1
tomlkit==0.11.8
ujson==5.7.0
whatthepatch==1.0.5
wrapt==1.15.0
yapf==0.33.0Could it be possible that ruff outputs something different from one invocation to the other?
A possibility is that I have multiple versions of ruff installed: the one in the virtual environment created by mason and the one installed from pipx. And I did indeed updated the one from pipx but I don't remember when (I thought it wouldn't matter but it might).
I'll try this hypothesis by reinstalling an older version of ruff with pipx.
Oh, I understand. In my case, it was referring to the globally installed ruff with pipx. (I don't remember when I installed it either TT)
Removing the pipx ruff solved the problem
I'll try this hypothesis by reinstalling an older version of ruff with pipx.
OK I did try with ruff==0.0.250 and the issue was back again. Removing ruff from pipx or updating it to the latest version fixed the issue.
Thanks for your help @jhossbach
Closing as solved.