WARNING: This project is written for educational purpose only. Use them at your own risk.
git_xy
helps to synchronize sub directories between git repositories
semi-automatically, and may generate pull requests on github
if
changes are detected on the destination repository. It works like
rsync
, but for git repositories.
git_xy
reads a list of source/destination specifications from a
configuration file, and for each of them, git_xy
fetches changes
from the source repository and synchronizes them to destination path
(thanks to rsync
). It finally generates commit and creates new PR
(pull request) if necessary.
See more details in How it works.
git_xy
is a Bash4 script. It requires some additional tools on system:
- GNU tools:
awk
,rsync
,bash
,git
,grep
,sed
- Github command tool for PR creation: https://github.com/cli/cli/releases
The main program git_xy
can be installed anywhere on your search path
$ sudo wget -O /usr/local/bin/git_xy \
https://github.com/icy/git_xy/raw/ng/git_xy.sh
$ sudo chmod 755 /usr/local/bin/git_xy
Configuration consists of source/destination specification in the following format:
src_repo src_branch src_path dst_repo dst_branch dest_path [pr_base] [rsync_options]
which reads in the order
src_repo
,src_branch
,src_path
: The source repository, branch and pathdst_repo
,dst_branch
,dst_path
: The destination repository, branch and path.pr_base
is optional and is used to specify where you want the PR arrives. By default, it's the upstream repository.rsync_options
(optional): Any options forrsync
command. The first option should start with-
.
See examples in git_xy.config-sample.txt.
git@github.com:icy/pacapt ng lib/ git@github.com:icy/pacapt master lib/
Now execute the script
GIT_XY_CONFIG="git_xy.config-sample.txt" ./git_xy.sh
the script will fetch changes in lib
directory from branch ng
in the pacapt
repository,
and update the same repository on another branch master
.
If changes are detected, a new branch will be created and/or
some pull request will be generated.
If you provide GIT_XY_CONFIG="-"
, the script will read from STDIN
.
These PRs are generated by using sample configuration file.
Generated by the latest version of the script:
- icyfork/pacapt#18 (File synchronization)
- icyfork/pacapt#17
- icyfork/pacapt#14
- icyfork/pacapt#15 (same configuration,
but when
GIT_XY_REVERSE=yes
)
Generated by some older versions of the script:
GIT_XY_ENV_FILE
: Where you define some configuration for your script. In this file you can define/do whatever you want before the script starts. Please note that you may want to useexport
to make your variables available to subsequent processes.GIT_XY_CONFIG
: Path to the configuration file. When it is empty and/or not specified, the valuegit_xy.config
is used. You can also use-
to specifystdin
as the input source of configurations.GIT_XY_PUSH_OPTIONS
: Options used bygit push
command when new commit is generated by the script. Default to empty string.GIT_XY_SET_OPTIONS
: Options used byset
command. Seeman set
for details. For example, if you want to turn on debug mode, useGIT_XY_SET_OPTIONS=-x
. Please use this variable with care; and please note that the option-u
and+e
will be always enforced.GIT_XY_HOOKS
: Specify a list of post-commit hooks. By default, it isgh
. See Hooks for detailsGIT_XY_REVERSE
: When the value isyes
, the synchronization direction is reversed (dst becomes src, src become dst). This is useful when you want to fetch something from the downstream. Thepr_base
option is not changed when the option isyes
.D_GIT_SYNC
: Where the script fetches remote repositories. It's a hard-coded string$HOME/.local/share/git_xy/
here$HOME
is your home folder.
A hook is a predefined method which would be executed when new commit is generated. A hook is expected to return zero code.
gh
: Generate a Github pull request when new commit is created.
Nothing magic, it's a wrapper of git clone, rsync and git commit
:)
Let's say we have configuration file
src_repo src_branch src_path dst_repo dst_branch dst_path
the script will do as below
- Create a clone of the
src_repo
in~/.local/share/git_xy/src_repo
(The actual folder name is a bit different to avoid some special characters in the user input.) - Check out the existing branch
src_branch
- Create a clone of the
dest_repo
in~/.local/share/git_xy/dst_repo
- Check out the existing branch
dst_branch
- Create new branch from
dst_branch
(if neccessary). This branch is specially used for PR creation. - Use
rsync
to synchronize the contents of thesrc_path
anddst_path
. On the local machine where the script runs, it's a variant of the commandrsync -ra SRC DST
hereSRC
is~/.local/share/git_xy/src_repo/src_path
andDST
is~/.local/share/git_xy/dst_repo/dst_path
- Generate new commit
- If specified, execute the hook to generate new pull request on Github
Well, it's so easy right? It's an automation support of your handy commands.
- Add tests and automation support for the project
- Provide a link to the original source
- More
POSIX
;) - Sometimes we only need to create a PR without generate any commit
- Re-use existing branch to generate new PR
Done
- Read configuration file from
STDIN
-
hook/gh
: Return zero if a PR already exists - Support file synchronization...
- Add option to reverse the synchronization (dst becomes src and vice versa)
- Better error reporting
- Handle
--delete
option - Create a hook script to create pull requests
- Gather multiple sub-folder in a single PR
- Re-use existing
git_xy
branch - Better hook to handle where PRs will be created
- Add some information from the last commit of the source repository
- Make sure the top/root directory is not used (we allow that)
- Allow a repository to update itself
There are many tools trying to solve the code-sharing problem:
-
git submodule
: Create pointers to some commit hashes on the upstream repositories, and check out the upstream repository as sub-directory of the current repository upon request. It's you and your team who watch changes on upstream and fetch them manually. More submodules to watch, more issues to handle. -
git subtree
: Quite similar togit submodule
, but it doesn't provide fragile pointer. Instead, it fetches all upstream commits and creates some merge points in the current repository. This way it's more stable thangit submodule
, when you always have a copy of the upstream code in your repository. You get what you don't really mean: duplication of commits, bigger size, confused/noisy commit messages, and you have to learn how to merge (really?) Good reading: https://www.atlassian.com/git/tutorials/git-subtree.Git-subtree original experimental project is found here https://github.com/apenwarr/git-subtree.
Looked like git submodule
requires you to understand C programming language,
while git subtree
is kind of Python which hides pointers from your laptop:D
- https://github.com/esrlabs/josh: Acting likes
nginx
-reverse proxy,josh
"combines the advantages of a monorepo with those of multirepo setups by leveraging a blazingly-fast, incremental, and reversible implementation of git history filtering". Discussing on HN: https://news.ycombinator.com/item?id=27844363 - https://github.com/zephyrproject-rtos/west:
This tool is origined from Zephyr-RTOS project, and inspired by
google repo tool and
git submodule
. "West lets you manage multiple Git repositories under a single directory using a single file, called the west manifest file, or manifest for short." By default, it uses Zephyr's manifest file (https://github.com/zephyrproject-rtos/zephyr/blob/master/west.yml) but it doesn't really care much, that means the tool can be used by other project too. I've found this tool on July 2nd 2021. I'll get back with some reviews. - https://github.com/ingydotnet/git-subrepo:
Another
git-slave
-liked project, which helps to manage multiple small repositories. It usesgit work-tree
, and it helps to generate pull/push/merge command in sub repositories by using the same command. Forget yourgit command
, as you have to learn to use this new wrapper for all sub commands (commit, pull, merge...) Written completely in Bash. - https://github.com/twosigma/git-meta:
Another
git-slave
which usesgit-submodule
to create the meta repository. Another NodeJs tool - https://github.com/mateodelnorte/meta:
Similar to
git-slave
, which creates a meta repository that includes multiple small repositories. You adapt both mono/micro repository idea. Written in NodeJs... - https://sourceforge.net/projects/gitslave/:
The project is still on
sourceforge
, looked like it's not maintained. The idea is to have meta project which handles multiple repository. The basic tutorial is here: http://gitslave.sourceforge.net/tutorial-basic.html.
- https://github.com/teambit/bit (npm only): Use this if your current Linux kernel is written in NodeJs.
- https://github.com/lerna/lerna (javascript only): Use this if your current Linux kernel is written in Javascript.
- https://gerrit.googlesource.com/git-repo/ (Android only?): Use this if your server is running on Android.
- https://github.com/microsoft/VFSForGit (sic, Windows only): Use this if you are running Linux inside a VM inside a Windows host.
- https://github.com/splitsh/lite: Split a repository to read-only standalone repositories
- https://github.com/unravelin/tomono:
You hate micro-
***
enough and you just want a big repository that includes all your small repositories. This tool helps you.
Well, there are too many tools... What I really need is a simple way to pull changes from some repository to another repository, generates some pull request for reviewing, and the downstream maintainer will decide what they would do next.
Morever, this process should be done automatically when the upstream
repository is updated. Human intervention is not the right way when
there are just 100 or 500 repositories because of the raise of the
micro-repository design
(if any) :D
- A monorepo misconception - atomic cross-project commits: https://www.snellman.net/blog/archive/2021-07-21-monorepo-atomic/, https://news.ycombinator.com/item?id=27903282
The script is written by Ky-Anh Huynh. The work is released under a MIT license.