Prevent extension from breaking when we can't parse Dark code

[StachuDotNet]

Our extension tries to provide syntax tokenization every time a .dark file is updated.

How syntax highlighting works

This is provided by two mechanisms, all managed in lsp-server:

• when you open/close/save a .dark file, our extension's language server keeps track of the updated content in a local cache. That cache is part of our LspState model. Here are the bits where we're listening to the client/editor for text document sync events, with docSync.dark doing most of the heavy lifting.
• whenever a .dark file is within view, in the editor, the client/editor sends a message to the server asking for semantic tokenization - the response to this request will be used in the editor to provide syntax highlighting. here is where we listen for and respon to that request. The meat of the logic relevant is here.

The problem

Specifically, look at the handleSemanticTokensRequest. Its process is:

1. get the doc from cache
2. parse it (docText |> Parser.parseToSimplifiedTree |> Parser.parseCliScript)
3. tokenize it -- basically, return a bunch of integers that tell the client/editor how the text should be tokenized

parsedFile
|> LanguageTools.SemanticTokens.ParsedFile.tokenize
|> EncodeSemanticTokens.toRelativeTokens
|> EncodeSemanticTokens.toLspFormat

I think step 2 in particular is failing - we probably succeed in tree-sitter 'parsing' to the 'simplified tree' but our Parser.parseCliScript fails poorly when it encounters anything it doesn't expect. Basically, we need some solution that has handleSemanticTokensRequest not fail when Parser.parseCliScript encounters something it doesn't expect.

Ideally, we highlight as much as we can parse, and report either nothing or problems on the bits we can't parse. It'd probably be good enough, even, to just not highlight anything when we can't parse, and report one big "we can't parse!" Diagnostic.

I'm not sure the best way to tackle this. Maybe Parser.parseCliScript should have a bunch of Results or something? Maybe we just need something to catch a failure somewhere? Not sure.

By "fail" I mean that some Dark code raises an exception. That exception is printed to stdout. Since all stdout is sent to the editor/client, the editor/client freaks out that we sent something that wasn't a valid JSON request/response/notification, and dies.

Things to keep in mind

• there's a little bug with the doc sync - if you 'run' the extension and a file is open by default, we don't get any doc sync events, and so the doc isn't in our cache. that means we don't have enough context to highlight. I'm not sure how to best deal with that, but in the meantime I've been working around this by closing the auto-opened file, and re-opening it. The reopening will send the docSync messages across the wire.
• all of the generic groundwork for supporting a language server protocol is in dark/packages/languageServerProtocol. It might be useful to peek at, but I don't think you'll have to change anything here. Our language server is in dark/packages/languageTools/lsp-server. There's another thing around there, lsp.dark, that should probably be deleted, as it's no longer used. (this was from the days when we had a node-based LSP server)
• the extension and the language server communicate with each other via stdout - basically a bunch of (json) text messages back and forth, sending notifications, requests, and responses.

---

Quick guide to testing the extension

• in the devcontainer, cd vscode-extension
• npm i should work there to install the extensions' dependencies
• hit F5 to run the extension; this will open up a new window
• the extension won't be "activated" until you actually navigate to a .dark file - I recommend opening user-code/darklang/scripts/sample-for-testing-extension.dark, which I've previously had success highlighting fully.
• logs reported by the language server should be in rundir/logs/lsp.log. We log: incoming notifications+requests, outgoing notifications+responses, and anything else you Builtin.printLine
• general CLI logs should be in rundir/logs/cli.log. When working with the LSP, you'll see a bunch of our language server's responses here, in JSON.

[StachuDotNet]

@OceanOak has completed this.