Any interest in collaboration?
Closed this issue · 18 comments
Hello!
I just stumbled onto this project. I work on LSP a lot myself, and was wondering if you have any interest in finding ways to collaborate. Of course, no pressure at all! But, at the very least we can be aware of each other 😃
https://github.com/ChimeHQ/LanguageServerProtocol
https://github.com/ChimeHQ/LanguageClient
Hello Matt!
I'm totally aware of your work, including your contributions to CodeEdit. You're sort of a legend in my world 😅
Back in the day, I tried intensely to leverage https://github.com/chimehq/SwiftLSPClient for this whole Codeface thing but couldn't get it to work or properly separate out the functionality that I required from it – which might just have been my inability.
I know your "eco system" has evolved quite a bit since, and I'm totally open to adopt it rather than re-inventing all this while ending up with less mature solutions. I just hadn't had the time or "trigger" to stop and re-evaluate. I'm moving very fast with Codeface, as my time window for it is open now and will mostly close end of March. So I'm a bit restless and couldn't "deal" with not being able to adjust something quickly to the requirements of this project ...
At the same time, I see wheels being re-invented all around, in particular with developer tools (including my own Swift packages) and would love to learn how to change that. So collaboration on these things is indeed the holy grail.
I'm curious about your thoughts.
Haha, well this is way too complimentary! I'm truly flattered.
I'm really sorry to hear you had trouble with SwiftLSPClient. Honestly, I did too at times. I'm not 100% sure just starting new repos was the smartest way to go, but that's what happened...
I've looked through your work just a tiny bit. Codeface seems really cool! I love seeing tools like this come up! Please don't get distracted with this as you work on it. That's more important!
I'm totally into finding a way to merge. I'm not 100% clear on your use of websockets. But, I think that adding additional data transport mechanisms shouldn't be a problem to support.
Pretty excited to talk more about this!
I'm totally into finding a way to merge. I'm not 100% clear on your use of websockets. But, I think that adding additional data transport mechanisms shouldn't be a problem to support.
Yes. Now I remember it had to do with my special requirement/want to communicate with LSP servers via WebSocket, which required a bit more composability. If you look at the 1st diagram here, you see my LSP.ServerCommunicationHandler (a.k.a. Server) takes an LSPServerConnection. That connection is just a class protocol. And LSP.WebSocketConnection conforms to that protocol so I can inject it into the LSP.ServerCommunicationHandler ...
As far as I understand, the LanguageServerProtocol package represents just the LSP independent of higher level concerns, so I should be able to use it.
However, merging those higher-level concerns with LanguageClient is likely more involved.
Ultimately, Codeface (any client?) wants to talk to an LSP server by just exchanging LSP messages, ideally via async functions, and not care how requests are matched to responses or where messages go to and come from. Right now In Codeface, using an LSP server via WebSocket boils down to calls like this:
let symbols = try await server.requestSymbols(in: fileUri)Can you tell me more about the WebSocket usage? I'm very curious! It is something many servers support natively? I assumed just stdio and tcp maybe?
You are right about how that LanguageServerProtocol library works. The communication stuff is separate, and I use that feature quite a bit actually.
It could be that using LanguageClient would be tricky, but I have a feeling it would not be as hard as you think. There's not a lot of abstraction over the core LSP features. I've been adding async support to it gradually, but there are still a few places where it is missing.
Can you tell me more about the WebSocket usage? I'm very curious! It is something many servers support natively? I assumed just stdio and tcp maybe?
Yes, the client that directly talks to LSP server executables does so exclusively via stdio. But that client is LSPService which runs a local webservice offering access to LSP servers via WebSocket. This diagram illustrates the idea.
Then the client of LSPService (in my case Codeface) wraps its LSP.WebSocketConnection in a sort of server abstraction – i.e. in LSP.ServerCommunicationHandler.
There's also LSPServiceKit – a little package helping Swift clients use LSPService. The diagram over there gives some more context.
But why the hell all this, you ask? 😁 The LSPService#why section really explains it best and succinctly. Certainly most tools don't need the WebSocket route, but the vision for Codeface includes stupid easy installation via the App Store.
It could be that using LanguageClient would be tricky, but I have a feeling it would not be as hard as you think.
And I suspect you're right. Also, the beauty of a protocol is, I can start by migrating LSPService to LanguageServerProtocol and then LanguageClient without effecting SwiftLSP, LSPServiceKit or Codeface. I'd love to contribute to the convergence of efforts, and this could be the first step. As long as I can still obtain individual LSP "Packets", LanguageClient doesn't even need to support the whole WebSocket thing, I think. But I can't promise when this "refactoring iteration" will come to pass.
I've been adding
asyncsupport to it gradually, but there are still a few places where it is missing.
I'm excited to discover how all that works in your packages. In SwiftLSP, all async requests rely on this one function:
func request(_ request: LSP.Request) async throws -> JSON { ... }Plus a generic convenience function that uses the one above:
func request<Value: Decodable>(_ request: LSP.Request) async throws -> Value {
try await self.request(request).decode()
}Ok, this all makes sense.
You may be surprised/shocked to learn that you can, in fact, run processes outside of the sandbox on macOS and also be App Store-compatible. I learned of the trick from another app that uses LSP on the store. You have to use an XPC service, which is allowed to be unsandboxed (at least today...)
I open sourced the basic concept I use to do this here: https://github.com/chimeHQ/ProcessService
Wow! I tried to answer the question 2 years ago, including asking in the Swift forums. The result I remember is: XPC services must be sandboxed when deployed in the App Store. The same rules apply. So it would just move the problem somewhere else. Quick research today suggests that hasn't changed, so 🤯
Now, your solution is incredible! It neither requires signing the executable nor bundling it with the main app. I'd love to adopt it and throw away LSPService since that would make things much easier for the user.
My only question is: To what degree and in what way might this "trick" bend the App Store review guidelines? I can't afford to refactor this foundation only to have the app rejected later ...
Oh, btw: Only this one line didn't compile, the rest compiled out of the box 😊
Gah, that stupid API is so hard to use, because Apple keeps changing the SDKs within Xcode, and there's no way to detect that at compile time. I pushed a change that I believe will fix that issue.
I pushed a change that I believe will fix that issue.
And indeed it does!
Haha, I see you spoke with Marcin Krzyzanowski in that thread. He's the one that figured this all out.
I really cannot speak about the App Store rules. All I can say is there are apps on the store today that are doing this. Given that you already have a solution built, I wouldn't make a change unless you really felt like there was a compelling reason.
Given that you already have a solution built, I wouldn't make a change unless you really felt like there was a compelling reason.
The compelling reason is, the user wouldn't have to discover, download, trust and run LSPService 😊 Thanks for this hint, I'll investigate it for sure. Might turn out to be super valuable!
Haha, right 😅
Well, I do hope it is valuable. Here's how I use it. It's a little bit hacked together, but I was just trying to finish quickly.
https://github.com/ChimeHQ/ChimeKit/blob/main/Sources/ChimeLSPAdapter/RemoteLanguageServer.swift
I don't love the code, but it works, and I had so many other things finish I just called it and moved on for now.
Thanks for the example code! I figure Chime 2.0 will actually also use that to host the LSP servers via the/a ChimeKit extensions(s).
Will there be one configurable LSP extension to rule them all, or will you have to wait for people to implement Chime extensions for every language? Because this is part of what I tried to address with LSPService for myself. I'm irritated that every LSP server needs to have its own VS Code extension, thinking "Isn't that why the LSP is called a Protocol? To provide a common standard?". I hope all servers can just be used via one universal configuration (path + arguments + environment), but maybe I'm still naive about this ...
Btw, no need to feel weird about "hacked" code. It's awesome how much of your code you share. Also the involved terms are familiar, and I'd be able to draw much value out of it 🙏🏻
Regarding this thread and collaboration: I've come away with lots of input, inspiration and TODOs – more than I can implement right away. If there's anything I can do for you, I'm happy to do so. And I'll surely get back to you when I embark on the migration to your Swift packages.
Thanks for the example code! I figure Chime 2.0 will actually also use that to host the LSP servers via the/a ChimeKit extensions(s).
Yes, but it is more complex. Because ExtensionKit extensions are outside of the main app, each one potentially needs its own copy of the XPC service to work. Originally, I was bundling it within ChimeKit, but that turned out to be problematic for many reasons.
I just moved it into its own specialized target in the ProcessService package. Which makes it even easier to use on its own!
Will there be one configurable LSP extension to rule them all, or will you have to wait for people to implement Chime extensions for every language?
Unfortunately, this is not technically possible for all LSP servers. Many deviate from the spec slightly, have custom features and/or bugs that require writing code to support. So, while you can just use the stock LSP client, there are many good reasons to customize the behavior on a per-server basis.
And it gets worse! Notably for Ruby, the server configuration and even which server you use is project- and environment-dependent. The best experience requires evaluating how the project is set up and customizing the experience. And influences which server is best.
However, Chime does take a stance regarding extensions and languages. In general, extensions should concern themselves with supporting a language. An LSP server to do that is an implementation detail. Swift is a prime example. Today, the Chime extension uses sourcekit-lsp. But, in the future, I hope to migrate towards using SourceKit directly, without LSP in between. We'll discourage making extensions that are server-specific.
I'm so glad this has been a productive conversation. I almost didn't even write to you! The number one thing I ask is you let me know if you run into problems!
It seems to me we're now closing in on a central challenge for tool development and a central opportunity for collaboration or at least for a much needed convergence in the space in general:
If every server needs its individual "care giver", the question arises whether they can find some common ground/interface on a higher level of abstraction. That is: Can your extension system be general-purpose enough to be employed by other developer tools?
It seems rather insane that every dev tool would solve this anew 😅 I'd love to "plug in" Chime extensions into Codeface and would also contribute to make that possible.
Heh. I'm pretty sure this would actually work today. ChimeKit contains all of the communications infrastructure needed. And, any app can adopt the extension point identifiers, or just incorporate the open source code directly and side-step ExtensionKit altogether.
So, yes, I think you could actually do this. And, you'd just need to bring in ChimeKit to do it. But, you'd have to implement the client and host APIs, and I have a feeling that will be difficult.
But, you'd have to implement the client and host APIs, and I have a feeling that will be difficult.
I'm sure there is a way to get a foot in the door via small iterative steps. For now I'll close this issue.
After the next little Codeface update, I'll do a deep dive into ChimeKit, ExtensionKit and XPC services. Then I will surely (have to) get back to you. And Vatsal has expressed great interest in going the same route, merging with- and leveraging your solutions. I'll approach all that in collaboration with him.
So excited! 😊