Monorepo
danez opened this issue · 9 comments
It is nice that snapdragon as well as micromatch are nicely separated into small pieces, but I really think that having all of them in separate repositories make maintaining them a big pain, especially when making a breaking change in one of them that other rely one and then trying to update the whole ecosystem.
I'm also maintaining some projects that use monorepos with yarn workspaces and lerna and it gives you the ability to have small packages, but makes bigger changes super easy as they are all in one repository and one PR. So in my opinion setting up a monorepo would be a nice addition for snapdragon and micromatch.
So I wonder what your opinions are on this? And also for the micromatch org.
I don't want to come into a new project and change everything :) but I think it could make things a lot easier for everyone.
lol, this is so coincidental. Honestly, earlier today I was wondering if someone would ask this.
I don't want to come into a new project and change everything :) but I think it could make things a lot easier for everyone.
Don't worry about that, the feedback is great. I want to have this dialog because I feel that I'm missing something about monorepos, and (if you hadn't created this issue) I was going to tweet to ask someone for feedback about it. I've tried working with monorepos, but I'm having a hard time seeing the benefits everyone describes, and I see many disadvantages that seem unimportant to others, and perhaps shouldn't really matter to me either.
If you're willing to share your experience with monorepos, can you explain a little more about the advantages and disadvantages you've seen?
First I'll give you my impression of monrepos so you can dispell any misunderstanding I might have.
My experience has been that once a monorepo is created, it ends up coupling the projects together completely. Even if that's not the original intent, it just happens over time. The individual projects often no longer have their own documentation, etc. Conversely, I think that many people expect the snapdragon and micromatch projects to be completely coupled, but I don't understand that perspective, as it's an anti-pattern when the goal is publishing self-encapsulated modules. IMHO, it seems like the only projects that would make sense for the monorepo format are ones that have dependencies that are not intended to be used (or can't pragmatically be used) as standalone projects. For example, these pug projects would be a good candidate for a monorepo. But in this situation, isn't the packages directory just a way of versioning all of the parts of the library that could have been in a lib or src folder? If, in a perfect universe, the individual packages were still documented and usable as standalone libraries, then I can see how the monorepo format would make sense. But rarely seems to be the case.
To that end, I can see how snapdragon might be a good candidate, but I don't see how micromatch would make sense as a monorepo. Each of the libraries in the micromatch org is a completely separate project that should stand on its own. They don't "exist for micromatch" specifically.
Thoughts?
I'd like to get @doowb's thoughts on this as well.
edit: I also wonder if it's worth considering that while it might be a bit harder to maintain projects separately and keep them decoupled, isn't that axiomatic of the module pattern? Rather than solving the "maintenance problem" by using monorepos, we've focused on creating the tools we need to make disparate projects easier to maintain. Just a thought.
Agree with both. Snapdragon is perfect candidate for monorepo, but not the micromatch - as Jon said, projects there are completely decoupled standalone ones and can be used separately, and micromatch just combines them. Same is for the @enquirer too.
As about @snapdragon, yes it may have benefits, but the thing that i don't like about monorepos is navigating through code and docs, also that it have single issue tracker.
I also tried to create monorepos few times, but ended up that i don't like it and it won't give me much - it will be more discomfortable to be than it gives some comfort. So i'm interested in hearing some thoughts from you @danez too.
I also agree with @jonschlinkert and @olstenlarck and I think it's mostly because of Jon's last comment which was added as an edit. We've built tools that make maintenance easier and keeping the separation isn't too bad.
To go along with @olstenlarck comment about documentation, there have been many times when I get an error in a module or want to look at the readme for a module and I type npm repo some-module and it goes to the root of a monorepo. When looking at the packages folder, I see this (not picking on babel, but this was my experience 😁):
Also, from my observations when making a change in a subpackage and publishing, the monorepos seem to require publishing all of the packages in the monorepo.
I think that monorepos are harder for new contributors to get involved. For someone that's making changes across many projects in the same dependency tree, I can see how monorepos can be an advantage but also see how it tends to couple the dependencies together. Because of these reasons, I think each package should have self-contained maintenance tools that can be run by new contributors, as well as, "power contributors". I think of this as like a vscode workspace or sublime text project. The IDE is the tool that a maintainer chooses to use to work on the code and group projects together. Then from some root directly where the projects are cloned into, they could choose to run a tool that executes the maintenance tools inside the projects.
Those are my 2¢
I'm currently trying to update snapdragon in micromatch, so that bundling works with webpack. That's where my idea about monorepos came from, because it is tedious work to go through all dependencies, update snapdragon, make everything work, release a new version got to next dependency and do the same thing again. Same thing seems to happen in snapdragon when someone would attempt to update snapdragon-node in snapdragon, which requires some other packages (capture, captureSet) to be update first. And as soon as that would be done we can update snapdragon in micromatch repos again too :)
In this case a monorepo would show it's advantage as I could do everything at once without having to manually link projects together or releasing projects just to be able to be able to update it in other projects.
My experience has been that once a monorepo is created, it ends up coupling the projects together completely. Even if that's not the original intent, it just happens over time. The individual projects often no longer have their own documentation, etc.
Monorepos in general do not force any specific structure in the packages or if versions of the packages should be independent or synced. The main focus of a monorepo is usually to make maintenance and development easier, which also includes removing development tools from each single package and moving it into the root of the monorepo. This makes it easier to make changes to these tools by updating their configs or updating to a newer version in one place. But it is still up to the maintainers to choose how to layout everything and if for example documentation should be in one place or separate in each package.
The improved maintenance workflow and being able to make breaking changes to one package and immediately seeing the impact on the other packages, is one of the reasons babel moves nearly everything into the monorepo. For example when babel still had its parser (which is also used a lot standalone btw) in a separate repository, new syntax additions would always required making a PR there, waiting till it is merged, waiting till a new version is released and then create PRs in the babel repo to update the parser version and support the new syntax their. Sometimes we lost contributors in the middle of the process and then we had half done stuff lying around. By moving it into the monorepo contributors can simply add one PR and do everything in there.
The biggest downside I see is that the issue trackers are not split by package, especially when there ~5 new issues per day it is not practical anymore to watch the repository and get email notifications for everything. This could potentially be solved with code owners, but I haven't tried that. (https://blog.github.com/2017-07-06-introducing-code-owners/)
I guess both models have disadvantages and advantages.
This could potentially be solved with code owners, but I haven't tried that.
how did I not know about that?! that's interesting! thanks for sharing it.
Based on your comments, I'm tempted to try the monorepo format for snapdragon. Give me some time to think about it, I'd like to play around with the monorepo tools and see what would change for us. Honestly, even if it's not my own personal preference, I see a pattern emerging around monorepos and big "ecosystem" projects, and I'd be remiss to ignore it.
and thank you for sharing your thoughts, I really appreciate it!
@danez, @jonschlinkert and I were talking about this and we'd like to get your feedback on how we could approach a monorepo that will allow us to do the following:
- keep the code in their own packages: the
lexer,parser,compiler, etc... - use the code in
snapdragonwithout requiring thedependencies. Basically, require them in to use them like normal, but inline the code so the published version ofsnapdragondoesn't have any dependencies.
Then I think we would do something similar with the micromatch organization where there's a monorepo for many of those projects but the code ends up being published as only having one dependency (snapdragon).
I hope this makes sense and I'd appreciate any feedback you have. Also, @jonschlinkert if there's something I left out or got wrong, please correct me.
@doowb I guess out of the box this is not really doable. Splitting packages in smaller ones comes with the cost of having more dependencies.
To work around this, snapdragon could be bundled with for example rollup into a single file. In the rollup config you can then define which dependencies should be inlined and which should be keept as is. This would then also make it necessary though to patch package.json on publish to remove the dependencies that are now inlined, because even though code lives in the same repository in a monorepo, the individual packages are still separated from each other and have to have proper dependencies on each other in the package.json in order to work.