Unicode Support

[toroidal-code]

This will require switching to a completely different lexer, as OCamllex cannot handle Unicode strings.

Alternatively, we can wait until the compiler is self-hosting, and then just build Unicode support in.

[zellio]

I say Unicode now or we'll never do it

Ideas: https://www.orcaware.com/pipermail/ocaml-pxp-users/2005-December/000057.html

[toroidal-code]

I agree.
We're going to go with ulex.
I'll make a feature branch and start working on this immediately.

Relevent Links:

• https://github.com/vbmithr/ocaml-sgf
• The Menhir Reference Manual

[toroidal-code]

Changed.

The introduction of ppx hooks and compile-time AST rewriting make it so that we don't need to deal with the mess that is introducing ocamlp4.

sedlex is much cleaner, and will be what we use.

A feature branch has been made, and I'm actively working on.