Syntax Files

Question

Syntax Files

oliver-sanders opened this issue 6 years ago · 23 comments

oliver-sanders commented 6 years ago

Provide lexers for the Cylc suite definition syntax in different editors / systems.

Currently supported:

vi - etc/syntax/cylc.vim
emacs - etc/syntax/cylc-mode.el
gedit - etc/syntax/cylc.lang
kate - etc/syntax/cylc.xml
pygments - cylc-sphinx-extensions

Planned support:

A few major editors support the TextMate grammar format:

TextMate (TextMate)
Atom (TextMate or Tree-sitter)
Visual Code Studio (TextMate)
Ace (TextMate)

Dropped support:

google code prettify

Feature support table:

🆗 - Full Support
⛔ - Not Supported
⛅ - Partial Support

Feature	Example	`vi`	`emacs`	`gedit`	`kate`	`pygments`
1. Basic INI/Cylc Syntax
1.1 Sections / Settings	`[a]b=c`	⛅ [3]	⛅ [5, 6]	⛅ [5]	⛅ [5]	🆗
1.2 Comments	`a=c # c`	🆗	🆗	🆗	🆗	🆗
1.3 Multiline Str	`""" ...`	🆗	🆗	🆗	🆗	🆗
1.4 Include Files	`%include`	⛅ [1]	⛔	⛅ [8]	⛔	🆗
2. Exotic Cylc Syntax
2.1 Graph tasks + deps	`foo`, `=>`	⛔ [14]	🆗 [12]	⛅ [9]	🆗	🆗
2.2 Graph Comments	`foo # Foo`	🆗	🆗	🆗	🆗	🆗
2.3 Graph operators	`&`, `()`, `!`	⛔	⛔	🆗	⛅	🆗
2.4 Inter-Suite	`<a.b::c>`	⛔	⛔	⛔	⛔	🆗
2.5 xtriggers	`@wall_clock`	⛔	⛔	⛔	⛔	🆗
2.6 Parameter	`<x>`	⛔	⛔	⛔	⛔	🆗
2.7 Parameter	`<x-1>`	⛔	⛔	🆗	⛔	🆗
2.8 Parameter	`<x=1>`	⛔	⛔	🆗	⛔	🆗
2.9 Qualifier	`a:fail`	⛅ [2]	⛔	⛔	⛔	🆗
2.10 Inter-cycle	`a[-P1D]`	🆗	🆗	⛔	⛔	⛅ [13]
3. Templating
3.1 Shebang	`#!...`	🆗	⛔	🆗	🆗	🆗
3.2 Jinja2	`{%`, `{{`, `{#`	🆗	🆗 ~~[7]~~	⛅ [11]	🆗	🆗
3.3 Jinja2 Multiline	`{% \n %}`	🆗	🆗	🆗	⛔	🆗
3.4 empy (basic)	`@X`, `@(`, `@[`	⛅ [4]	⛔	⛔	⛔	🆗
3.5 empy (full)	`@?`, `@!`, `@...`	⛔	⛔	⛔	⛔	⛔

[1] Can't handle quoted file names.
[2] Problem with custom qualifiers containing underscores
[3] Equals characters in setting values breaks highlighting
[4] Parenthesis in expressions can cause issues
[5] Comments after sections break highlighting
[6] => symbols are highlighted outside of graph strings
~~[7] Cylc comments following Jinja2 blocks don't display as comments~~[Second observer: I don't see this issue?]
[8] Unquoted files aren't highlighted
[9] foo=>bar displays incorrectly
[10] Suicide triggers not highlighted
[11] Jinja2 comments not displayed as comments
[12] => highlighted, & tasks appear in the same mode as e.g. standard values under sections but
that seems appropriate?
[13] Arbitrary whitespace not supported in arthymatic (e.g. [^+P1D] is fine but [^ + P1D] is not).
[14] => acknowledged in triple-, but not single-, quoted graph strings.

Please update the table as syntax capabilities change.

2018-10-05 #2784
- emacs
  - Support multiline Jinja2 statements
  - Fix scoping issues
2019-01-07 metomi/rose#2274
- pygments
  - Support inter-suite triggers
  - Support include files
  - Support xtriggers
  - Support empy
  - Support arthritic in inter-cycle triggers
  - Fix line continuation chars in headings

Answer 1 · 2018-08-14T09:52:26.000Z

I volunteer myself as emacs rep! In the interests of achieving equality between the two most common editors, I am adding addressing Jinja2 Multiline (first & foremost) & EmPy to my To Do list.

(Oh & +1 for creative emoji use.)

Answer 2 · 2018-08-14T10:02:53.000Z

@sadielbartholomew - I was an emacs user once, and then ...

Answer 3 · 2018-08-14T10:04:47.000Z

(apologies 🐑-ish 😁)

Answer 4 · 2018-08-14T10:13:07.000Z

That made me 'laugh out loud', though I don't think I will be having such a revelatory moment (with vim certainly... maybe with Atom)!

Answer 5 · 2018-08-14T10:14:14.000Z

To be fair it did take me a little while to see the light!

Answer 6 · 2018-08-14T10:21:37.000Z

IDEs aside the top text editors from the StackOverflow survey:

Answer 7 · 2018-08-14T11:02:55.000Z

...okay so I was completely wrong about emacs & vim being the two dominant editors (I got this impression but admittedly never thought to investigate)! Those are interesting stats indeed.

Answer 8 · 2018-08-14T20:41:25.000Z

There's also MS Visual Studio Code, a standalone code editor built on electron. It's pretty slick and has a pretty comprehensive vim emulation mode. I tried it a while ago, but I felt a bit "dirty", and I still like the lightness of vim.

https://en.wikipedia.org/wiki/Visual_Studio_Code

Answer 9 · 2018-08-15T07:48:04.000Z

I have also tried VSCode recently (on my home Windows box and an Ubuntu VM). It is pretty good. Not sure if I can actually use it for real productivity work or not.

Answer 10 · 2018-08-15T10:07:22.000Z

Added to the list, I thought it was an IDE (been a while since I used visual studio).

Answer 11 · 2018-08-15T12:54:13.000Z

I've come across EmPy highlighting for Atom, maybe it will be useful:
https://github.com/dirk-thomas/language-empy

Answer 12 · 2018-11-01T14:32:05.000Z

(Updated main table in leading comment to reflect emacs mode update from merge of #2784.)

Answer 13 · 2020-02-11T20:24:57.000Z

n.b. I have started playing with with textMate for VSCode, and have got the basic bits working.

Answer 14 · 2020-03-07T10:28:52.000Z

Tim, as (now!) a VSCode user, who's just been hunting around a bit re options for rose/cylc, give me a shout if extra (noob-level) support on looking into textMate/being pilot user/reviewing docs/other helpful!
(have seen Oliver's recommendation elsewhere that one can ~use ini syntax rules for rose).

Answer 15 · 2020-03-26T15:07:51.000Z

Just to announce I've picked up on Tim's TextMate grammar for VSCode, should be ready to go in a few days time. As discussed in the last couple of Metomi video meetings, we'll need to figure out how we're going to host it, GitHub-repo-wise, considering it'll also be ported over to Atom

Answer 16 · 2020-04-08T19:06:40.000Z

The VSCode/TextMate grammar is in an almost deliverable state.

Feature	Example	`VSCode/TextMate`
1. Basic INI/Cylc Syntax
1.1 Sections / Settings	`[a]b=c`	🆗
1.2 Comments	`a=c # c`	🆗
1.3 Multiline Str	`""" ...`	🆗
1.4 Include Files	`%include`	🆗
2. Exotic Cylc Syntax
2.1 Graph tasks + deps	`foo`, `=>`	🆗
2.2 Graph Comments	`foo # Foo`	🆗
2.3 Graph operators	`&`, `()`, `!`	🆗
2.4 Inter-Suite	`<a.b::c>`	🆗
2.5 xtriggers	`@wall_clock`	🆗
2.6 Parameter	`<x>`	🆗
2.7 Parameter	`<x-1>`	🆗
2.8 Parameter	`<x=1>`	🆗
2.9 Qualifier	`a:fail`	🆗
2.10 Inter-cycle	`a[-P1D]`	🆗
3. Templating
3.1 Shebang	`#!...`	⛔
3.2 Jinja2	`{%`, `{{`, `{#`	🆗 [1,2]
3.3 Jinja2 Multiline	`{% \n %}`	🆗 [1,2]
3.4 empy (basic)	`@X`, `@(`, `@[`	⛔
3.5 empy (full)	`@?`, `@!`, `@...`	⛔

4. Extra
4.1 Cylc8 graph Syntax	`[[graph]]R1=...`	🆗
4.2 Shell script	`script = echo "foo"`	⛔
5. ISO 8601
5.1 Intervals	`P1Y1M1DT1H1M1S`	🆗
5.2 Datetime long	`2000-01-01T00:00:00-11:30`	🆗
5.2 Datetime short	`20051201T0215+0530`	🆗
5.3 Illegal intervals	`P1D1Y`, `P1H`	⛅
5.4 Illegal Datetime	`2000-01-01T0000`	⛅

[1] Enabled even if #!Jinja2 shebang not present
[2] ~~Jinja2 content all uniform (no further highlighting within Jinja2 blocks)~~ Can be used with a 3rd-party Jinja2 extension to provide Jinja2 highlighting inside cylc files

I've done it using a JavaScript file that can get 'compiled' to the JSON grammar file (used by VSCode and Atom). This makes it easier to reuse things when you have several similar rules.

The question is, how are we going to host it? I think we'll need separate repos for the VSCode and Atom versions of the grammar. How about an independent main repo for just the core TextMate grammar files (js, JSON and build scripts)? The VSCode and Atom repos could be modified clones and get merged into from the main repo, for new releases?

In any case, we could create the VSCode repo for now - should I create an empty repo then do a PR with all the commits so far?

Answer 17 · 2020-04-08T21:42:59.000Z

The grammar ( and entire vscode plugin) are static so this isn’t really the domain of package managers like npm.

I think this might just be the kind of problem git Submodules we’re created to solve.

We would - Create a repo with the text mate extension in and a bunch of other repos for vscode, atom, heck maybe even textmate itself which use the same git Submodule.

Thoughts?

Answer 18 · 2020-04-09T09:24:43.000Z

I hadn't heard of submodules before but that looks promising

Answer 19 · 2020-04-09T10:54:48.000Z

Here is my VSCode repo for now. The Jinja2 support is currently in a separate branch to master

Answer 20 · 2020-04-14T10:09:01.000Z

I had a go at opening about 6 suites randomly picked from <site users path>/*/cylc-run/*/suite.rc when and they all looked Good. Tried it with one (I'll tell you which on a private channel) which I picked out as likely to push the envelope, and there are some crispy bits of jinja2 around the edges, but by-and-large it was ok.

Answer 21 · 2020-04-15T22:38:16.000Z

Note our syntax support is far from perfect for any editor (I still use Vim mostly) so I woudn't worry too much about "the crispy bits" (nicely put @wxtim 😁 )

Answer 22 · 2020-04-16T08:25:22.000Z

Agreed, just document the deficiencies in the table above, this is something we may get people in the Cylc community working on.

Answer 23 · 2020-05-04T15:56:52.000Z

(Posting here in addition to Discourse)

The extension for VSCode has been released: Visual Studio Marketplace; GitHub repo

Plus TextMate bundle for TextMate, PyCharm, WebStorm & Sublime Text 3: GitHub repo

Update: the package for Atom has been released: https://atom.io/packages/language-cylc