Bring back support for other language plugins

[steinybot]

It seems that in the transition to the new Rust compiler we lost support for compiler language plugins. This is very sad. For a long time we have been using a language plugin to generate Scala.js code. Is there any way to bring this capability back?

[steinybot]

We might be able to workaround this by:

1. Having our build tool take our Scala files and output the inline definitions to .graphql files and create Scala.js facades that correspond to the JavaScript that relay-compiler generates.
2. Pass those .graphql files to relay-compiler and have it generate JavaScript.

I haven't figured out the last step which seems the easiest. It seems from the docs that relay-compiler expects the input language to be the same as the output. Can it not take the input directly from .graphql files?

[captbaritone]

Can it not take the input directly from .graphql files?

We do not currently and I don't expect we will since a significant portion of our value proposition comes form encouraging the pattern of colocating your fragment with your UI module.

cc @zth who's experience with ReScript Relay might be relevant here

[zth]

@steinybot in order to get anywhere meaningful with this you're going to have to fork the compiler and add your own language mode using Rust. That's what I've done with ReScript. It's quite a lot of work, and I've had to resort to a few "hacks", mainly because the compiler is built to emit TS or Flow, which are really similar and shares most of its syntax with only very minor differences. ReScript needs are similar but not similar enough for the current printer setup to work.

I would however love to see some refactoring in the type generation of the compiler to make plugging in other languages easier. Last time I looked at this I think my conclusion was that it'd be a lot of work, and it's questionable what parts we could actually land given that it wouldn't benefit Meta anything, so they're unlikely to be able to spend the resources needed for testing/reviewing to actually land it. But things might have changed since.

Brain dump of what the main challenges are for plugging in other languages:

The whole type generation pipeline is designed only for TS and Flow, so language plugin type printers hook in at a very late stage, and only get access to an AST that's made solely for emitting TS/Flow. If we instead could get access to some AST before that stage, which has more type information and is a structure rather than just a printer for individual (and very specific/low level) AST nodes, I think life would be a lot easier. As it is now, with RescriptRelay, I reconstruct an AST like that in reverse from the printable AST I get fed in the printer. I then print myself using that structure. So, it's a bit cumbersome.

[steinybot]

Fortunately I found a way to use sbt, a build tool for Scala, to do what we need. The sbt plugin is in https://github.com/goodcover/scala-relay/tree/jason/gc-3120-find-a-way-to-compile-gql-without-a-compiler-plugin. The code needs a good refactor but the main parts are there.

What I do is:

1. Use Scalameta to parse the Scala source files and extract the inline GraphQL definitions into .graphql files.
2. Use the GraphQL parser from Caliban to parse the GraphQL definitions and then write out Scala.js facades that correspond to the JavaScript produced by relay-compiler.
3. Wrap all the extracted .graphql files and any additional files with the graphql tagged template and write those out as .js files. I have to do this because relay-compiler only seems to reliably transform executable definitions if they are in the same source language as the target.
4. Run relay-compiler as you would for a normal JavaScript project except there the JavaScript sources solely contain the graphql tagged templates.

It wouldn't make sense for our team to learn and maintain a Rust project. Rust is nice and all but it is solving a problem we don't have. With a good caching strategy you only recompile a very few number of definitions per change, unless of course you are changing the schema, and that does not happen often enough to justify the maintenance overhead. With the approach above we can stay within the Scala ecosystem and the speed seems very good without any optimisations.

The downside is that we have to write quite a lot of code ourselves and replicate a lot of stuff that relay-compiler does. We used to get a lot of this from the previous transformers. Things like determining variables and generating the corresponding query for refetchable fragments are especially painful.

If relay-compiler could emit a more general AST as described by @zth then that would be ideal, especially if it contained information about what changed. It doesn't need to be in process or even IPC. Using files is totally fine, after all the input and output is already files so no need to over-optimise and over-complicate.

I understand that this is probably never going to happen.