Shopsys Monorepo Tools

Tools for building and splitting monolithic repository from existing packages. You can read about pros and cons of monorepo approach on the Shopsys Framework Blog.

We created these scripts because we couldn't find a tool that would keep the git history of subpackages unchanged.

You may need to update your git (tested on 2.16.1).

This repository is maintained by shopsys/shopsys monorepo, information about changes are in monorepo CHANGELOG.md.

Quick start

1. Download

First download this repository so you can use the tools (eg. into ~/monorepo-tools).

git clone https://github.com/shopsys/monorepo-tools ~/monorepo-tools

2. Preparing an empty repository with added remotes

You have to create a new git repository for your monorepo and add all your existing packages as remotes. You can add as many remotes as you want.

In this example we will prepare 3 packages from github for merging into monorepo.

git init
git remote add main-repository http://github.com/vendor/main-repository.git
git remote add package-alpha http://github.com/vendor/alpha.git
git remote add package-beta http://github.com/vendor/beta.git
git fetch --all --no-tags

3. Building the monorepo

Then you can build your monorepo using monorepo_build.sh. Just list the names of all your previously added remotes as arguments. Optionally you can specify a directory where the repository will be located by providing <remote-name>:<subdirectory>, otherwise remote name will be used.

The command will rewrite history of all mentioned repositories as if they were developed in separate subdirectories.

Only branches master will be merged together, other branches will be kept only from first package to avoid possible branch name conflicts.

~/monorepo-tools/monorepo_build.sh \
    main-repository package-alpha:packages/alpha package-beta:packages/beta

This may take a while, depending on the size of your repositories.

Now your master branch should contain all packages in separate directories. For our example it would mean:

main-repository/ - contains repository vendor/main-repository
packages/
- alpha/ - contains repository vendor/alpha
- beta/ - contains repository vendor/beta

4. Splitting into original repositories

You should develop all your packages in this repository from now on.

When you made your changes and would like to update the original repositories use monorepo_split.sh with the same arguments as before.

~/monorepo-tools/monorepo_split.sh \
    main-repository package-alpha:packages/alpha package-beta:packages/beta

This will push all relevant changes into all of your remotes. It will split and push your master branch along with all tags you added in this repository. Other branches are not pushed.

It may again take a while, depending on the size of your monorepo.

Note:
The commits in the split repositories should be identical to those from the original repo, keeping the git history intact. Thus, if you have checked out the original master previously, you should be able to fast-forward to the new version after splitting.
The only known exception is a signed commit (note that GitHub signs commits made via its web UI by default). If you have signed commits in your original repository, the split commits will NOT be signed. This will prevent monorepo_split.sh from pushing the unsigned commits to the remote.
To overcome this you can add the --force flag to the git push calls in the script, but it may cause unforeseen consequences if you're not sure what you're doing.

Add a new package into the monorepo

When you have the monorepo, you may find a reason for adding a new package after some time you already use the monorepo. In this case, don't use monorepo_build.sh, but do following steps:

Create a new repository, for example, vendor/gamma
Add remote into the monorepo git remote add package-gamma http://github.com/vendor/gamma.git
Create a new directory in the monorepo packages/gamma
Add the code and commit it

Use split tool with the new package

~/monorepo-tools/monorepo_split.sh \
    main-repository package-alpha:packages/alpha package-beta:packages/beta package-gamma:packages/gamma

Reference

This is just a short description and usage of all the tools in the package. For detailed information go to the scripts themselves and read the comments.

Contributing

Thank you for your contributions to Shopsys Monorepo Tools package. Together we are making Shopsys Framework better.

This repository is READ-ONLY. If you want to report issues and/or send pull requests, please use the main Shopsys repository.

Please, check our Contribution Guide before contributing.

Support

What to do when you are in troubles or need some help? Best way is to contact us on our Slack http://slack.shopsys-framework.com/

If you want to report issues, please use the main Shopsys repository.

Saik0s/monorepo-tools

Shopsys Monorepo Tools

Quick start

1. Download

2. Preparing an empty repository with added remotes

3. Building the monorepo

4. Splitting into original repositories

Add a new package into the monorepo

Reference

monorepo_build.sh

monorepo_split.sh

monorepo_add.sh

rewrite_history_into.sh

rewrite_history_from.sh

original_refs_restore.sh

original_refs_wipe.sh

load_branches_from_remote.sh

tag_refs_backup.sh

tag_refs_move_to_original.sh

Contributing

Support