A new Branching Model for the Release Process

[mickael-menu]

We got a few discussions regarding the state of develop and the release processes, here's a proposal to answer some of the points raised:

• Reading apps should be able to integrate cutting-edge features and bug fixes without waiting for the next release.
• Releases should be able to be published regularly, regardless of current development efforts.
• Releases should not be made on any arbitrary develop commits.

TL;DR

We need to agree beforehand to a list of small independent feature sets (basically a roadmap).

• develop is always buildable, but may contain regressions and unstable APIs.
  • Feature PRs are made in this branch.
  • Once develop contains a complete implementation of the currently developed feature sets, it is merged into next-release.
  • It could be used to build alpha versions.
• next-release contains the stable feature sets available in the next release, but may contain regressions.
  • Bug fix PRs are made in this branch, before being merged back into develop.
  • It could be used to build beta versions.
• master is either removed because tags are sufficient, or renamed stable.

Bief Introduction to a Release Process

Since the releases are not in sync with the development efforts, it's dangerous to start a release process from an arbitrary commit in develop. Instead, we could agree on a list of feature sets to implement together which will be used as "release checkpoints". This way, releases will always be made from a stable point API-wise.

Here are a few examples of feature sets:

• Audiobook navigator.
• Support for byte range requests.
• Implementation of the Publication Encapsulation proposal.

A release checkpoint contains related feature sets which might influence each other's APIs. For example "Audiobook navigator" + "Support for byte range requests".

Bug fixes are integrated continuously and should not block a release.

Upgrading third-party dependencies is a critical step that should not be done just before making a release, because we need time to make sure they are not introducing issues.

Branching Model

To support this process, the current develop/master branches are not sufficient, because development is blocked until a release is done.

develop (alpha)

The develop branch contains in-progress feature sets.

• It must always be buildable, ideally checked through CI (although it is not working at the moment for mobile).
• PRs made on this branch should only contain features, not bug fixes.
• Bugs might occur and in-progress APIs might change from one commit to the other, therefore it is not recommended to use develop in production apps.

Unfortunately, we don't have the resources to thoroughly test develop every time a PR is merged. But maybe we can agree on a small list of features to be tested manually after each merge?

Once develop is merged into next-release, it's a good time to upgrade the dependencies of the project in develop when relevant, to make sure they are sufficiently tested.

next-release (beta)

The next-release branch is used to prepare the next release.

• Bug fixes (of already released features) are made in next-release, before being merged back into develop.
• Once a given release checkpoint is completed in develop, it is merged into next-release (through a dedicated summary PR).
  • This is a good point to perform a bit more testing, such as creating highlights. We could create a list of features to test at this point.
• This branch is the default one setup in the Git repositories.

The release manager is free to start the release process when suited (either scheduled or improvised), because this branch is always ready for it. It might only contain bug fixes for minor versions, or feature sets for major versions.

Compared to a tagged release, next-release might still contain regressions, therefore it should be considered as a beta version for production apps. It's acceptable to use if you have additional in-house testing processes. A big advantage over develop is that we get bug fixes quickly, without the instability of in-progress features and API changes.

master

You probably heard about the current debate in the community regarding master, in the wake of Black Lives Matter. Whether or not we agree, one could argue that master as used right now is pointless, since HEAD is always pointing to the latest tag. Developers using released versions are usually doing it through a package manager and not submodules, since they handle semantic versioning.

Therefore, I suggest ditching master. If not, we can also rename it stable.

[aferditamuriqi]

The way i see this, this should be split into two different topic, as the release process is not just a simple branch name. There is a lot more to it. and at the end i don't really have any preference which branch i use for my release process, as long as that branch is untouched while in process.

This topic here should only be about branch naming. and if you want to ditch branches go ahead. a day or two ago in slack it was said shouldn't implementers use master if they want a stable branch?, now we want to ditch master, fine too, we just have to make a decision.

overall there needs to be a stable branch, because we are not keeping all the release branches around. or no stable at all?

so i am fine with if you want to change how branches are named and used. but about the release process, i'd rather have a different thread, discussion etc.

[mickael-menu]

@aferditamuriqi I actually agree with you, sorry if that was ambiguous, I didn't mean to document the whole release process. I just set up a bit of context to make sense of this next-release branch. I'm trying to find a solution for what's impacting my work right now.

This topic here should only be about branch naming. and if you want to ditch branches go ahead. a day or two ago in slack it was said "shouldn't implementers use master if they want a stable branch?", now we want to ditch master, fine too, we just have to make a decision.

That was because we only had develop and master. But in this proposal, implementers would use a semantic versioned tag for stable versions, and next-release for recent bug fixes and beta features.

overall there needs to be a stable branch, because we are not keeping all the release branches around. or no stable at all?

The next-release branch is actually "permanent", so it will always be around and can be used in a submodule. It's basically a more stable develop. However, if someone wants to use a well-tested version, they should use a package manager and semantic versioned tags.

That being said, I'm actually fine keeping master around. I just took this opportunity to think a bit more about it with BLM and the fact that technically it's not very useful. (Is anybody using master in a submodule?)

I'm not trying to force push this solution, I'll be more confident if you're comfortable with this branching strategy as well.

[mickael-menu]

Regarding the names, I understand it might be a bit controversial and maybe even non-standard (although I think it fits git-flow). Instead, we could also have the same principle, but with develop being used as the next release branch, and introduce a alpha branch which will contain the inter-dependent features being developed at the moment.

The point of this is to have smaller, easier to manage PRs while being able to work on several features at the same time. (I know it's not ideal, but we have some requirements from our respective projects and I found it almost impossible to follow a linear roadmap).

[llemeurfr]

It's personal, but I'd really be in favor of the process corresponding to Mickaël's last comment, much more standard IMO.

[aferditamuriqi]

So basically the exact same what i recommended last week just in different words.

https://readium.slack.com/archives/C703MSTQU/p1592583813121300

[aferditamuriqi]

there needs to be a stable branch (currently master, we can rename it to stable)
there needs to be a stable develop branch (currently develop)
there needs to be a unstable branch (this is the branch that you are referring to before it goes into any stable)

2 options:

1. stable, develop, unstable
2. stable, release, develop (while release would actually be misleading)

i would prefer 1. since it will leave the branches as they are and just introduce a new branch before develop, this is backwards compatible with everyone out there already using develop (as we know everyone is using in submodule this branch including you Mickael)

Basically we can rename master to stable, leave develop as is, and branch out an unstable branch from develop.

[aferditamuriqi]

If you do not prefer 1.

it's just a matter of naming, then second option:

2. stable, release-candidate, develop (next-release or just release is very misleading)

[mickael-menu]

So basically the exact same what i recommended last week just in different words.

I said it in the introduction, but this proposal spawns from our discussions on Slack, to have a more "formal" and documented agreement. You would have been credited as a co-author if it was an actual proposal (which it may become once the release process is documented as well).

I'll edit the issue to reflect the names.

[mickael-menu]

@aferditamuriqi I missed your more recent posts. Thanks that's exactly the kind of feedback I wanted.

So I prefer 1., except for unstable. How about alpha?

And what about the details/boundaries between these branches that I described in the issue?

[aferditamuriqi]

@aferditamuriqi I missed your more recent posts. Thanks that's exactly the kind of feedback I wanted.

So I prefer 1., except for unstable. How about alpha?

perfectly ok to name it alpha, i was just throwing in some names to better see the puzzle pieces

[mickael-menu]

Great, I'll update the issue.

However I still think we should remove master/stable. Let's talk about it on a dev call.

[llemeurfr]

I know more and more people have an issue with "master" these days and I heard Github wants to rename "master" to "main" https://www.theregister.com/2020/06/15/github_replaces_master_with_main/. But nobody is speaking about deprecating the use of develop

So first question: is renaming "master" a move related to the current trend? if yes why not using the "main" term?
Second question : why renaming "develop" to something else nodoby will have heard of?

Git flow creates a release branch per release (good illustration here), but an issue is that these release branches are dead branches later. We had long discussions before about adopting git flow or not and decided not -> using tags on master/main to keep track of releases.

Conclusion / proposal =
3. main, develop + feature branches or an "alpha-vX" branch when there is a need to have a multi-feature WIP branch (which is not so often; I bet next year nobody will need it anymore).

[aferditamuriqi]

I agree with that, i actually didn't propose a renaming of develop, but rather keep develop as is, and any feature/alpha/etc branches are the ones that are merged into develop then but those are the branches that are worked in.

Quite a while back we had agreed on having feature/branches and fixes/branches which pretty much corresponds with the way we had/have been working.

renaming branches is perfectly ok, to fit in the current circumstances, but leave develop as is would be my preference

obsolete branches should not be kept around, just because they were used for a feature or a release or a tag. we will end up in chaos if to many branches are kept

[aferditamuriqi]

Great, I'll update the issue.

However I still think we should remove master/stable. Let's talk about it on a dev call.

we do need a main/stable or whatever branch because we have to have one branch that actually has the latest tagged, reasons why: in the develop branch for example the tags and dependencies are in progress, and there will always be a timeframe where that will be the case. the develop branch will be in progress for a certain amount of time,

which branch do we point anyone that is using submodules? we can't just say "don't use submodules" . implementers will use submodules, dependency managements in different flavors etc. lcp can only be used as submodule.

i can't come up of any reason that support getting rid of a branch , and then later actually have to create multiple in between branches. we can't solve everything with one branch.

[mickael-menu]

@llemeurfr

So first question: is renaming "master" a move related to the current trend? if yes why not using the "main" term?

Not really, but I suggest we let the dust settle to see what the industry will choose before renaming. Personally I feel main only makes sense with in the context of GitHub Flow which has no develop branch, but only a single main (master) branch, and a number of feature branches. In my opinion stable has a clearer intent in our case.

Second question : why renaming "develop" to something else nodoby will have heard of?

There might be a misunderstanding, I don't think Aferdita or I suggested that. Re-reading the initial propal it does look like alpha sounds like develop now, I will revert back the change to the original name for the sake of clarity.

Git flow creates a release branch per release (good illustration here), but an issue is that these release branches are dead branches later. We had long discussions before about adopting git flow or not and decided not -> using tags on master/main to keep track of releases.

That was actually the inspiration for my initial proposal, but I simplified things a bit. Instead of transient release branches, the permanent next-release would be used to prepare the next release. This way no dead branches. Tags are limited because they don't allow to continue development between two stable states of the codebase, and release branches are useful to do hotfixes without pulling all the new changes from develop.

@aferditamuriqi

which branch do we point anyone that is using submodules? we can't just say "don't use submodules" . implementers will use submodules, dependency managements in different flavors etc. lcp can only be used as submodule.

I think people don't use master (with HEAD always pointing to the last released tag) with submodules as it would be a bad practice. If you want to use stable software, you would use semantic versions (usually with a PM or manually with tags) and therefore update every minor versions, and when you're ready carefully upgrade to the next major one.

But I'm fine to keep it around anyway if you prefer.

[mickael-menu]

I feel like we're going round in circles here.

This discussion started because of the need to do a release, while features were half-baked in develop, which is not really compatible. We either need to have a branch pre-develop (like alpha-x) where we can work on features which need more than one PRs and might be inter-dependent, or post-develop (like next-release) which reflect a stable version of develop and can be used as a starting point for a release.

Conclusion / proposal =
3. main, develop + feature branches or an "alpha-vX" branch when there is a need to have a multi-feature WIP branch (which is not so often; I bet next year nobody will need it anymore).

I think that fits the bill, and you may be right about the fact that this problem is only temporary. There have been a lot of architectural changes lately, but it won't always be like that.

[aferditamuriqi]

which branch do we point anyone that is using submodules? we can't just say "don't use submodules" . implementers will use submodules, dependency managements in different flavors etc. lcp can only be used as submodule.

I think people don't use master (with HEAD always pointing to the last released tag) with submodules as it would be a bad practice. If you want to use stable software, you would use semantic versions (usually with a PM or manually with tags) and therefore update every minor versions, and when you're ready carefully upgrade to the next major one.

confused again, yes people use submodules, and use them based on develop, or master for that fact. either way, what i am saying is, that we can't just make assumptions or exclusions. i am trying to see the harm of keeping a good and stable branch around. i can't see any harm or keeping but i see issues by not keeping. our decisions are really only around renaming that particular branch, not about getting rid of, and the develop branch again as is, and this new alpha before develop branch as you described above.

i think we are all on the same page here, but in just different ways of understanding.

[aferditamuriqi]

bottom line, is , i need one branch (currently that was develop) that i can prepare and handle release from . and i need one branch which is the resulting of the final release tags (currently that was master). the develop branch has always a period of time, 1 to 3 weeks or so, in which the release process happens, within that time the only stable branch is the (current master)

[mickael-menu]

i think we are all on the same page here, but in just different ways of understanding.

I think so too. So let's keep everything as-is for now (and maybe revisit renaming master once we see what's happening with GitHub)? I will use a alpha branch to contain any on-going feature changes until it's more stable, but do bug fixes directly in develop. And later on maybe we'll be able to get rid of alpha.