Move lexical model compiler to keyman repo

[mcdurdin]

So. A few things:

1. The merge of the model info should warn if there are mismatching version numbers, and ...
2. The version number shouldn't be in the .model_info, and ...
3. The canonical data is always in the source files, not the .model_info, because ...
4. Lexical models can be built, stored and distributed outside the lexical-models repository and CI chain. Those lexical models won't have a .model_info file, because that is used only for the CI deployment. It is important we don't end up building too many dependencies on the repository structure by accident because it is likely, if we are successful, that this repo will get very large. The intent is for models in this repo to be of high release-level quality and we generally want to discourage experimental models here, because models here can be automatically installed by Keyman. This leads me to conclude that ...
5. The compiler needs to be moved out of the lexical-models repo and into the keyman repo, and the .model_info management needs to be clearly delineated. Now, the compiler is part of the Keyman Developer toolchain. So I think what we need to do is update the build process to pull the latest stable (or $tier) version of the compiler from downloads.keyman.com (so that's something that needs to be added to the Keyman Developer CI). We could register an NPM package but I don't think we are quite ready to do that; happy to defer to wiser heads though. For consistency, then ...
6. This same process needs to be done with the keyboards repo -- so we don't include the kmcomp compiler in the repo, rather pull it at first build (or at appropriate times) from downloads.keyman.com.

Originally posted by @mcdurdin in #14 (comment)

[eddieantonio]

I'm starting to feel the pain of leaving this unaddressed. I find myself needing to make changes both to the LMLayer (keymanapp/keyman) and the compiler at the same time. I think these codebases belong in the same repo. Proposal:

1. Move tools/ to keymanapp/keyman/common/lexical-model-compiler.
2. Have this repo have a dependency in package.json to install the lexical model compiler, as npm install github:keymanapp/keyman#master. We don't need to publish an NPM package!
3. In this repo, have the build.sh automatically checkout the latest version of the compiler. Call the compiler, like kmlmc (Keyman Lexical Model Compiler) from this repo.

That's it! Thoughts @mcdurdin?

[mcdurdin]

1. Move tools/ to keymanapp/keyman/common/lexical-model-compiler.

I think probably keymanapp/keyman/developer/lexical-model-compiler is more appropriate because it's not something that is shared between platforms. Long term, I want to move Keyman Developer into /developer anyway and split it from Keyman Desktop, moving shared code into common/windows because I think that will improve our code organisation and maintenance (#1763).

2. Have this repo have a dependency in package.json to install the lexical model compiler, as npm install github:keymanapp/keyman#master. We don't need to publish an NPM package!

My main concerns here are:

a. we are pulling a huge repo as a dependency for a tiny compiler component
b. there are plenty of other components in Keyman, in particular KeymanWeb itself, which are appropriate candidates for this model of installation; it's a little confusing (and overkill) to use the same pattern for all of them.
c. I'm not clear on how well this fits into our alpha/beta/stable development cycle. I'm open to changes in how we manage our development lifecycle but we need to be careful :)

What if we instead npm install https://downloads.keyman.com/developer/{tier}/{version}/{kmlmc.tgz}? I would also consider updating downloads.keyman.com to allow us to refer to https://downloads.keyman.com/developer/{tier}/latest/{kmlmc.tgz} where {tier} is one of stable, beta, alpha.

Very open to better ways of doing this. Just not sure that a dependency on the whole keyman repo is going to serve us well.

[mcdurdin]

I find myself needing to make changes both to the LMLayer (keymanapp/keyman) and the compiler at the same time.

One last comment: I really believe the compiler should be maintained in keyman repo, not lexical-models, so no argument there.

However in terms of multiple related PRs targeting multiple repos, I'm less bothered. Multi-repo changes are a hassle with GitHub because PRs are so much a single-repo / single-branch concept, but we frequently need to do multi-repo changes with Keyman anyway (e.g. website, keyboard and keyman), or multi-branch changes (e.g. publish a fix to alpha and stable), and so I am "comfortable" with the idea of doing multiple PRs :)

I am working on moving the compiler over now -- appreciate any additional thoughts on architecture per previous comment.

[mcdurdin]

This has been largely addressed by #31 and keymanapp/keyman#1766, so I am going to go ahead and close this issue. If any other side issues are still present, let's open them as separate issues.