Highlighting sometimes doesn't start until x lines in the file
elandorr opened this issue · 11 comments
Current ST and addon versions.
It's loaded properly, as the folding works, but doesn't highlight. On this newly made test it doesn't seem to ever highlight. On other documents it starts after a few lines.
What's haunting it?
Edit: I just randomly noticed there is a difference in the syntax chosen:
- Markdown
- Markdown | nothing, not even your colorscheme
- MultiMarkdown | this one seems to do the trick
- MarkdownEditing
- Markdown | nothing, not even your colorscheme
- MultiMarkdown | colorscheme, but doesn't highlight properly as per this bug report
The one from your own menu is broken, but the default runs with it. Weird.
Not choosing your addon makes it work:
Never experienced such an effect. Syntax definitions are tested against ST 3211, 4107 and latest dev build. None of them failed like that, too.
What ST version are you using?
4126 with the 3.1.3, should both be the current versions.
What's with the choices available in the first place? I'd expected the plugin to simply offer a 3rd option, itself.
Can't reproduce with ST 4126 and MDE 3.1.3 on Windows.
Choises? Do you mean the color scheme chooser quick panel, which pops up after installation?
When placing caret/cursor at a non-highlighted line and hitting ctrl+shift+P what does the scope name popup display?
I mean the choices I posted above:
Of those 4 only the one I said works.
Ctrl-Shift-P opens the the command palette, where is the scope supposed to be seen?
The scope name popup can be opened via ctrl+shift+alt+p, sorry.
The menu item Markdown represents ST's default Markdown syntax package.
MarkdownEditing provides syntax definitions from this package, which are much more complete and follow CommonMark specifications more closely than those shipped with ST up to build 4131.
What about MultiMarkdown? So MDE is supposed to enhance both?
The scope in the 3 not working ones looks like this:
And in the working one as you expect:
MultiMarkdown is basically the same as Markdown at the moment, but with support for its own frontmatter style.
Found the culprit. Will be fixed soon.
Cool, thank you for your effort! Mind sharing a 1line tldr what caused it? ST is a bit of a blackbox to me. (MDE is so far the best MD highlighter, MD is surprisingly completely broken on vim! Not one vim plugin that can even stick to basic specs. Is MD so surprisingly tricky to parse?)
The issue you see is caused by a missing pattern in frontmatter context of MultiMarkdown.sublime-syntax.
The following lines mean: First parse everything within frontmatter and continue with markdown when done.
MarkdownEditing/syntaxes/MultiMarkdown.sublime-syntax
Lines 15 to 21 in 72a23cd
The current implementation expects a frontmatter to be present.
MD is surprisingly completely broken on vim
I am not surprised about that. Most syntax definitions of VIM I've seen so far are rather naive and simple.
Is MD so surprisingly tricky to parse?)
Short answer: Yes.
Markdown expects at least two runs for tokenization (finding words to highlight):
- look for container boundaries (e.g.: begin/end of block quotes, paragraphs etc.)
- parse inline data
This results in certain priorities. The parser in run 2 can't look beyound boundaries of a context specified by run 1 for instance. That's what ST's syntax engine needs to immitate in a single run.
I can't say anything about VIM, but many older syntax engines such as TextMate 1.0 format (tmLanguage) just iterate through and array of regexp patterns to search for content to highlight. So it's very tricky up to impossible to immitate required algorithms for certain languages.
ST's syntax engine is a pushdown automaton which also tokenizes text in a single run, but with the ability to switch context. It means it can enable/disable certain sets of regexp patterns depending on search results.
It's still tricky to parse especially lazy syntaxes like markdown this way. Especially as most regexp patterns are limited to a single line of content. This is what's called "line blindness" sometimes. The ways around that cause complexity to rise significantly.
To make it even more tricky many markups such as italics or bold are valid only if terminated correctly and can span multiple lines. Combined with ST's line blindness, it's nearly impossible to handle according to the specs. ST4's syntax engine provides some new features (e.g.: branching) to hopefully solve those issues, but it's even more complicated and time consuming to implement. You really need a deep understanding of both the syntax and the syntax engine to get that right.
Thank you for the details @deathaxe! Way out of my league, but interesting. I had to find MD parsers for various languages before, and while there are dozens, very few get it right. Impressive that you put in the effort for ST!
MD is a problem-format with all the offshoots too. Once you get used to something, you're stuck with incompatible files. And your favorite editor may not be the one with the best MD support. While Typora was in free beta it sure was enjoyable to have pretty much every extension ever in it. I guess sticking with GH flavor is reasonable long-term.
Fixed by 3.1.4