Newline issues on Windows

[bryphe]

Hi there,

Thanks for the excellent work you've done on the language server! Overall, it's working great on OSX and Linux - I'm working on incorporating it into Oni: https://github.com/extr0py/oni

There is one blocking issue on Windows, however:

Issue: The LSP protocol expects there to be \r\n\r\n following the Content-Header. This works as expected on OSX and Linux. However, on Windows, we actually get \r\r\n\r\r\n, which some LSP clients will not handle this. vscode-jsonrpc is strict on matching this and therefore never realizes the stream is complete, and never completes initialization. I've tested this using the stdio strategy - I haven't validated with the tcp strategy.

Defect: It turns out Python has some 'magic' behavior with newlines - see https://stackoverflow.com/questions/2536545/how-to-write-unix-end-of-line-characters-in-windows-using-python. It looks like Python is converting \n on Windows to os.linesep - which is \r\n.

This is the impacted code (server.py):

    def _write_message(self, msg):
        body = json.dumps(msg, separators=(",", ":"))
        content_length = len(body)
        response = (
            "Content-Length: {}\r\n"
            "Content-Type: application/vscode-jsonrpc; charset=utf8\r\n\r\n"
            "{}".format(content_length, body)
        )
        self.wfile.write(response)
        self.wfile.flush()

In this case, the \n in the \r\n is getting expanded - so we actually end up with \r\n.

Proposed Fix: The os.linesep should be checked. If it is \n, we should use \r\n as the line ending to conform to the protocol. If it is \r\n, that means \n will be expanded to \r\n, so we should use \n as the terminator above.

I'm not an expert in Python, so there may be a cleaner way to fix this. It looks like when reading from a file, there are options in terms of handling newlines, so maybe there is an option to set when writing to the output. I'm not sure as well if this would cause problems with the tcp server.

Let me know if this isn't clear - happy to give more info.

[gatesn]

I think the cleaner fix here (and it fixes #25 as well) is to figure out how to open stdin/out in binary mode!

Then we avoid any magic, as well as being able to count bytes not characters from the stream. Any ideas how to do that?

[bryphe]

Good point, working with stdin/stdout does seem like a more robust fix. Unfortunately, I'm not an expert in Python - I see this stackoverflow thread: https://stackoverflow.com/questions/908331/how-to-write-binary-data-in-stdout-in-python-3

But not sure if it works for Python 2 as well, and unfortunately it wouldn't solve the other issue.

[gatesn]

@extr0py I don't have access to a windows machine unfortunately, would you mind testing if this PR fixes your problem? #63

[bryphe]

Sure thing. I just tested on Windows 10 from the ngatges/binary-stdio branch. Working great for me!

Thanks for the fix!