Better Highlight with Neovim Treesitter
EduardoAraujoB opened this issue · 34 comments
Problem
Neovim 0.5 comes with treesitter, a plugin that improves a lot the syntax highlighting as described in the repository, but by using it with higlight enabled, I got weird higlights for some groups in TSX/JSX (in JS/TS files it works fine)
What happened
What I expected to happen
on my vscode, I got this
Machine Info
- NVIM v0.5.0-dev
- OS: Ubuntu 20.04
- Terminal: Tilix with Tmux
- TERM: xterm-256color
Additional Info
Well, there's always been disparity between VS Code and (Neo)Vim because of the different syntax engines. I too hope that tree-sitters in neovim will bring an improvement to this. (Aside: do you know if anyone has put the work in for vim to coordinate tree-sitters or similar with text-properties?)
At any rate, it's not clear what the issue is—are you saying that tree-sitter and our colorscheme are somehow clashing? We define highlight groups, no syntax stuff, so I'm not sure how that could happen.
@benknoble treesitter comes with new highlight groups which vim doesn't recognize, the problem that I'm having is to make dracula nvim work properly with this new groups on TSX/JSX, currently it doesn't show different higlights for variables, functions and JSX props, when the highlight is enabled by doing this lua require'nvim-treesitter.configs'.setup { highlight = { enable = true } } on my init.vim, it changes the way that nvim handles highlight groups, enabling that blue color for type definitions, but for some reason it doesn't work properly on the other cases that I showed, because the colorscheme doesn't give full support for nvim treesitter
I think I understand. The simplest fix is probably to continue to add more highlighting groups (see the section in colors/dracula.vim under has('nvim') with the Lsp* stuff). Feel free to submit a PR, or point us towards more resources on how to implement this.
Also just FYI: I personally don't have any plans to support this until this feature makes it to an actual release.
Tree-sitter is currently only available in nightly and I don't think it's wise to get too carried away supporting nightly build features as they are subject to change at any time.
@dsifford Neovim nightly is planned to release soon, but if it doesn't make sense for the project by now, I can just close the issue
I think it's fine to leave the issue be as the "tree-sitter integration tracker." But it may not get much progress until the official release comes (which I had also heard was soon).
neovim 0.5 released today, I told that it was soon 🙂
Based on https://neovim.io/doc/user/treesitter.html#lua-treesitter-highlight, the syntax feature is still experimental?
And it looks like you use TreeSitter queries to map parts of the buffer to highlight groups.
I'm happy to support more "standard" groups for NeoVim's tree-sitter, but I don't see any documented there. Of course, I'm no expert on the feature. It would be helpful for someone who understands it to take point on improvements here. That leaves us in "if someone is willing to work on it" territory, unfortunately.
I'm looking into this a bit further now that 0.5.0 has landed.
Looks like the way to get tree-sitter syntax as of right now is via https://github.com/nvim-treesitter/nvim-treesitter
Let me kick the tires a bit on this and then we can prob get something together for it. Likely after the holiday weekend.
Ok I've played around with it for the last day or two and I'm not sure I feel comfortable right this moment supporting it specifically because nvim-treesitter is the entity right now that dictates highlight groups.
When the feature fully lands in neovim, these highlight groups may or may not be translated. So it feels premature to build up support for these groups in the meantime.
That all said, it's pretty darn close by default to the correct scheme. There's just edge cases sprinkled throughout.
Yeah, that plugin (?) now says experimental until 0.6.0. There is some built-in support, but maybe it's not "the whole package"?
I'm not sure about when and how treesitter will be included in neovim by default, but about the edge cases, some of them I figured out that it's related to parsing issues, anyway, if it doesn't make sense for the project right now, feel free to close the issue and we can open a new one when treesitter gets more stable
As far as I can tell, the highlight groups that would need to be supported are documented here. It seems like adding more groups to this existing block would probably solve the issue, at least judging by other color schemes that claim to support treesitter.
I'd be happy to help out on this issue, though I wouldn't know how to choose the appropriate color for each group. Is there some "standard" example of dracula syntax highlighting that we would need to mirror?
Edit: Now I see the default linking mentioned above.
Thanks @adriantrunzo, that's really useful information to have
@adriantrunzo you can use how vscode set the colors groups as a mental model and of course that there is differences between both of them, but the mainly goal is to define the color groups for values (which is usually in #6272a4), the color groups for functions (usually #50fa7b), reserved keywords (#ff79c6), function parameters (#ffb86c), comments (#6272a4).
also the full color palette is here https://draculatheme.com/contribute
@EduardoAraujoB Thanks, this spec document is exactly the information I was looking for. And, as you mentioned, I can also compare it to VSCode. I have a branch of my configuration files with treesitter and the treesitter playground configured, so I am going to take a look at highlighting this weekend.
After getting to use tree-sitter in neovim with various files this afternoon, I actually think that there is nothing to do here. Despite being an experimental/beta feature with highlight groups that might change, the dracula theme already supports tree-sitter quite well. As @dsifford mentioned, the coloring is quite close to correct and there are only a few minor differences when compared to VSCode (explainable, see below). The nvim-treesitter library already links all of its highlight groups to existing, traditional vim syntax highlight groups, which dracula defines. The dracula plugin only needs to redefine the tree-sitter highlight groups when the the defaults don't match the spec, and two previous pull requests already completed most of that work.
(It is worth noting that the nvim-treesitter repository maintains a page of themes that support it. There is an imposter dracula there.)
In my testing I did not notice any difference between highlighting for typescript and javascript. @EduardoAraujoB If you update all of your plugins do you still see a difference? I have also come across a few unresolved discussions about TS/JS issues in nvim-treesitter.
Moving forward, I think any coloring differences folks continue to notice will sometimes be fixed by changing an nvim-treesittter default (example), sometimes be resolved by an tree-sitter or nvim-treesitter issue, but more likely will be due to the nature of treesitter itself.
To that last point, if anyone is curious, I will present an example using Javascript/JSX, since that is the language presented in this issue. Note that I took the Neovim screenshot with the tag attribute patch linked above.
Neovim/nvim-treesitter:
VSCode:
It is important to remember that treesitter, while a great improvement over regular expressions, is similar to regular expressions in that it does not semantically evaluate the code. It parses the code into a tree based upon a syntax grammar and the highlighting comes from queries made against the nodes of that tree (you could think of the queries like regular expressions against the nodes, complete with capture groups and all).
In the screenshots above, the first difference we see is that the constant counter is highlighted in purple by VSCode as per the spec, but not by tree-sitter. While you could certainly write a query to correctly highlight counter via tree-sitter on that line*, the query wouldn't be of any use in the rest of the code. The next time counter appears in the code, tree-sitter has no mechanism for associating it with the previous query to highlight it as a constant, it can only apply syntactical labels, in this case identifier or a generic variable. Instead of confusingly having two different colors for counter, the tree-sitter javascript parser tags constants by just looking for all capital letters (a regex!)
For similar reasons, you can explain the remaining differences in the screenshot:
- The parser has no way of knowing that
useStatereturns a function in the second position, so it cannot tagsetCounteras a function (green). - The parser has no way of knowing that the second appearance of
currentinhandleClickis actually referring to the function parameter (orange). You could write a query to match both, but just like a regular expression it wouldn't generalize well. - etc
The semantic information necessary to resolve all of these differences will only come when neovim and the language servers implement the semantic highlighting part of the LSP spec.
Also fun to know, GitHub uses tree-sitter for highlighting code blocks, so you can see that is also suffers from the same limitations:
let notAConstant = 1
let NOT_A_CONSTANT = 1
const isAConstant = 1
const IS_A_CONSTANT = 1*: At the tree-sitter playground, input the following code and query:
const myVariable = { innerConst: 10 }
const { innerConst } = myVariable
const { innerConst: otherConst } = myVariable
const [ arrConst ] = [ 10 ]
console.log(myVariable)
(lexical_declaration
"const"
(variable_declarator
name: [
(identifier) @constant
(object_pattern
(shorthand_property_identifier_pattern) @constant
)
(object_pattern
(pair_pattern
key: (property_identifier)
value: (identifier) @constant
)
)
(array_pattern
(identifier) @constant
)
]
value: (_)
)
)
In short--occasionally we'll need minor fixes but otherwise we're already doing the best we can.
And I cannot wait for semantic highlighting. I know some servers already support it. It's really a matter of someone (???) doing the work to integrate it. Unfortunately for me, neovim will probably get it first, and I plan on sticking with vim for the time-being.
thanks a lot for your answer @adriantrunzo , it's working quite well, some of the problems that I had I figured out that it's related to treesitter itself and not the color scheme, but, I guess that it's not the case for this one
maybe it's related to a parser issue, but I tested it with tokyonight.nvim and got this
ok, I saw that the #255 solves the same issue that I have, I have been using it for some weeks and the other problems that I saw is not related to the colorscheme itself (I guess), so the treesitter support is good as well, @benknoble if for some reason you change your mind and want to try neovim, I can help, as far as I can tell it is a way better than vim in lsp, color schemes, etc...
I am not really familiar with how to write a theme for (n)vim but i thought i'd share a link to how Nord theme added the support for it: https://github.com/crispgm/nord-vim/blob/develop/colors/nord.vim#L306
Thanks @SlyBouhafs , I think that's pretty similar to what we're currently doing.
I've found how doom-one implemented highlights for treesitter and it's really pretty similar also https://github.com/romgrk/doom-one.vim/blob/master/colors/doom-one.vim#L540
dracula does not support neovim?
dracula does not support neovim?
@DartMitai I'm not sure from where you drew that conclusion, but dracula does indeed support neovim.
I've been using dracula with neovim and treesitter close to a year right now and the major issues that I had was already solved, it's working pretty good for me, I'm not sure if there is anything to do about it in this project, thanks everyone for the help
if there isn't more problems related I could close the issue as well
This is still pretty much needed. Dracula currently dosen't support treesitter.
@EduardoAraujoB Perhaps we can close this issue and encourage folks to open new ones for specific problems that they find with the syntax highlighting via treesitter? I think we've detailed pretty well here how this project does support the treesitter highlight groups, though, undoubtedly there will be specific cases to improve upon. Especially now that neovim 0.6 has been released, even more people will start to use treesitter.
@peri4n Would you mind opening a new issue for the particular problem you are running into, please? For example, include a text sample and/or screenshots along with the information necessary to reproduce (eg neovim version, OS, etc). I (and others here) would be happy to take a look when we have a free moment 😄 .
This is still pretty much needed. Dracula currently dosen't support treesitter.
@peri4n Yes, it does: from #253 (comment)
The nvim-treesitter library already links all of its highlight groups to existing, traditional vim syntax highlight groups, which dracula defines. [links omitted]
I'm getting to be more in favor of closing this as @adriantrunzo mentions. What are your thoughts @dsifford ?
@EduardoAraujoB Perhaps we can close this issue and encourage folks to open new ones for specific problems that they find with the syntax highlighting via treesitter? I think we've detailed pretty well here how this project does support the treesitter highlight groups, though, undoubtedly there will be specific cases to improve upon. Especially now that neovim 0.6 has been released, even more people will start to use treesitter.
I've been using neovim 0.6 since the 0.5 was released, now I'm using neovim 0.7 (I'm not sure when my OS updated my neovim to 0.7) and it is still working pretty good
I also agree about closing this and let peoples to open new issues
Agree. Let's close this for now. I encourage anyone to open specific issues related to tree-sitter highlighting if you run into something (it's far from perfect at present).
Hi, for anyone landing on this issue, maybe you'll be happy with this configuration override in your dracula setup:
dracula.setup({
overrides = {
TSTagAttribute = { fg = dracula.colors().green },
},
})