Tagging previous versions?

[gpdf]

I understand from conversations during the recent Northern Autumn IVOA InterOp, and from private communications, that this repository (postgrespro/pgsphere) is considered the most authoritative currently available source for the pgsphere extension, as far as the IVOA community is concerned, and the one most closely aligned with current developments in geometry-query support (e.g., for RegTAP 1.2). For this community, at least, it supersedes, I gather, pgsphere/pgsphere and akorotkov/pgsphere (neither of which have a visible fork relationship with the present repo).

This repo doesn't currently have any visible tags, let alone releases. Is that something we could remedy? There is some evidence in commit messages of some intent for there to be notional releases identified as "1.1.4.916", "1.1.5", "1.1.5beta4gavo", and "1.2.0".

Is there any value in trying to create some retrospective tags? Is the head of master stable enough that it is worth tagging?

Beyond that, should we be discussing means of producing release artifacts that DBAs could use to add the pgsphere extension more easily to deployed database servers?

This is an issue at the moment because Rubin is trying to bring up pgsphere-based TAP services for commissioning data (in addition to the main Qserv-based TAP services used in Data Preview 0) - it really isn't obvious how to determine what a known stable release of pgsphere might be, or to obtain a recent build from, e.g., Debian package service.

@msdemlei and @pdowler, at least, are likely to be interested in this, but we'd be grateful for advice from any quarter.

[gpdf]

Oh, I just discovered this: https://salsa.debian.org/postgresql/pgsphere , with a fair bit of recent activity. Very superficially it looks like it might just be minimal maintenance based on the very old v1.1.1?

[msdemlei]

On Mon, Nov 07, 2022 at 10:05:51PM -0800, gpdf wrote: Oh, I just discovered this: https://salsa.debian.org/postgresql/pgsphere , with a fair bit of recent activity. Very superficially it looks like it might just be minimal maintenance based on the very old v1.1.1?

No, the Debian package is supposed to track postgrespro (and hopefully does; at least the MOC functionality is in there). And certainly it would be great to have releases and tags, yes (not to mention some activity on the PRs, if I may say so myself...).

[obartunov]

Hello to all of you ! I am professional astronomer from Sternberg Astronomical Institute (Lomonosov Moscow State University) and work part-time in Postgres Professional company, so we actually use pgsphere in all our projects and understand its value for astronomy.

On Tue, Nov 8, 2022 at 8:35 AM gpdf ***@***.***> wrote: I understand from conversations during the recent Northern Autumn IVOA InterOp, and from private communications, that this repository (postgrespro/pgsphere) is considered the most authoritative currently available source for the pgsphere extension, as far as the IVOA community is concerned, and the one most closely aligned with current developments in geometry-query support (e.g., for RegTAP 1.2). For this community, at least, it supersedes, I gather, pgsphere/pgsphere and akorotkov/pgsphere (neither of which have a visible fork relationship with the present repo). This repo doesn't currently have any visible tags, let alone releases. Is that something we could remedy? There is some evidence in commit messages of some intent for there to be notional releases identified as "1.1.4.916", "1.1.5", "1.1.5beta4gavo", and "1.2.0". Is there any value in trying to create some retrospective tags? Is the head of master stable enough that it is worth tagging

Beyond that, should we be discussing means of producing release artifacts

that DBAs could use to add the pgsphere extension more easily to deployed database servers? This is an issue at the moment because Rubin is trying to bring up pgsphere-based TAP services for commissioning data (in addition to the main Qserv-based TAP services used in Data Preview 0) - it really isn't obvious how to determine what a known stable release of pgsphere might be, or to obtain a recent build from, e.g., Debian package service. @msdemlei <https://github.com/msdemlei> and @pdowler <https://github.com/pdowler>, at least, are likely to be interested in this, but we'd be grateful for advice from any quarter.

We decided to move pgsphere to our standard workflow with tagging, versioning, packages, testing and benchmarking. We are open for suggestions, especially, we need a test suite. Also, I want to be able to compile --without-healpix, which is currently included by default, do we really need healpix support by default ? I have a problem compiling healpix on my mac. Btw, what's about naming pgsphere-X.Y, pgsphere-healpix-X.Y, where X - is a major, Y - minor ? Best regards, Oleg

…

— Reply to this email directly, view it on GitHub <#9>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQURYWBUIXPJRBJFJB3UITWHHRDJANCNFSM6AAAAAARZ5MJ3Y> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

[msdemlei]

On Wed, Nov 09, 2022 at 09:12:15AM -0800, Oleg Bartunov wrote: On Tue, Nov 8, 2022 at 8:35 AM gpdf ***@***.***> wrote: > Is there any value in trying to create some retrospective tags? Is the > head of master stable enough that it is worth tagging > Beyond that, should we be discussing means of producing release artifacts that DBAs could use to add the pgsphere extension more easily to deployed database servers?

Postgresql.org build pgsphere and has that at least in their APT repo. If their other builds have it, too, that'd be plenty fine for me.

We decided to move pgsphere to our standard workflow with tagging, versioning, packages, testing and benchmarking. We are open for suggestions, especially, we need a test suite. Also, I want to be able to

What would you want in addition to what the current make test does?

compile --without-healpix, which is currently included by default, do we really need healpix support by default ? I have a problem compiling healpix on my mac.

HEALPix and MOC are *really* nice, and it would be a shame to throw it out just because of build problems; I'd rather fix those -- it's just C++, and you certainly don't want to be in a state where Mac folks can't use HEALPix. Having said that, there is a superior implementation of the HEALPix primitives in rust, https://github.com/cds-astro/cds-healpix-rust/. It would be *excellent* if we could (optionally?) migrate pgsphere to use that.

Btw, what's about naming pgsphere-X.Y, pgsphere-healpix-X.Y, where X - is a major, Y - minor ?

Major-minor sounds reasonable.

[obartunov]

On Thu, Nov 10, 2022 at 11:13 AM msdemlei ***@***.***> wrote: On Wed, Nov 09, 2022 at 09:12:15AM -0800, Oleg Bartunov wrote: > On Tue, Nov 8, 2022 at 8:35 AM gpdf ***@***.***> wrote: > > Is there any value in trying to create some retrospective tags? Is the > > head of master stable enough that it is worth tagging > > > Beyond that, should we be discussing means of producing release artifacts > that DBAs could use to add the pgsphere extension more easily to deployed > database servers? Postgresql.org build pgsphere and has that at least in their APT repo. If their other builds have it, too, that'd be plenty fine for me.

Honestly, I don't know about this build.

> We decided to move pgsphere to our standard workflow with tagging, > versioning, packages, testing and benchmarking. We are open for > suggestions, especially, we need a test suite. Also, I want to be able to What would you want in addition to what the current make test does?

Our workflow includes testing on different architectures, OS. I know at least one bug still exists in current pgsphere akorotkov/pgsphere#11

> compile --without-healpix, which is currently included by default, do we > really need healpix support by default ? I have a problem compiling healpix > on my mac. HEALPix and MOC are *really* nice, and it would be a shame to throw it out just because of build problems; I'd rather fix those -- it's just C++, and you certainly don't want to be in a state where Mac folks can't use HEALPix.

I don't mind excluding them, just to add an option to not include, since not everybody needs them.

…

Having said that, there is a superior implementation of the HEALPix primitives in rust, https://github.com/cds-astro/cds-healpix-rust/. It would be *excellent* if we could (optionally?) migrate pgsphere to use that. > Btw, what's about naming pgsphere-X.Y, pgsphere-healpix-X.Y, where X - is a > major, Y - minor ? Major-minor sounds reasonable. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQURYV433LOYJW3PV3UU33WHSVCXANCNFSM6AAAAAARZ5MJ3Y> . You are receiving this because you commented.Message ID: ***@***.***>

-- Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

[msdemlei]

On Thu, Nov 10, 2022 at 12:22:07AM -0800, Oleg Bartunov wrote: On Thu, Nov 10, 2022 at 11:13 AM msdemlei ***@***.***> wrote: > What would you want in addition to what the current make test does? Our workflow includes testing on different architectures, OS. I know at least one bug still exists in current pgsphere akorotkov/pgsphere#11

This sounds like an interesting one -- in particular since it would seem what's returned is something like uninitialised memory. But of course we'll first need a reproducer, because on my box everything looks just fine...

> > compile --without-healpix, which is currently included by default, do we > > really need healpix support by default ? I have a problem compiling > healpix > > on my mac. > > HEALPix and MOC are *really* nice, and it would be a shame to throw > it out just because of build problems; I'd rather fix those -- it's > just C++, and you certainly don't want to be in a state where Mac > folks can't use HEALPix. > > I don't mind excluding them, just to add an option to not include, since not everybody needs them.

An opt-out option (probably makefile variable) would be fine for me. Opt-in I'd like a whole lot less; let's try and avoid a situation where what I think should be the standard configuration require action beyond just make && make install.

[vitcpp]

Dear All, let me please refresh the discussion concerning versioning.

I propose:

• In general, we should follow some principles described in https://semver.org/
• Major number is for API incompatible changes.
• Minor number is for API backward compatible changes (old user sql should work with newest pgsphere).
• Patch number is for bug fixes (API is untouched).
• Mark each version with annotated tag like v1.2.0

What I'm concerned, now we increment only the third number (patch number) for those changes that include API changes. I guess we should increment minor number in this case. But if we decide to increment minor number on each API small change this number may become too big.

One more question - should we increment minor or patch number each time when merging a new PR or introduce release plan?

Another issue is an optional compilation of some parts like healpix. I do not exclude that some new optional compilation will appear in the near future. I think it should be the user responsibility how to compile. The version number should not reflect how pgsphere is compiled. I see the only issue - upgrade scripts. I guess we may implement some query function that returns the list of compiled features. It may help upgrade script to decide what to do.

I appreciate any comments, thank you!

[esabol]

This all seems reasonable to me.

One more question - should we increment minor or patch number each time when merging a new PR or introduce release plan?

1. If a PR adds to or changes the API, then that PR should change the major or minor version number.
2. If a PR does not change the API in anyway and does not change the output of any of the regression tests, I don't see a need to increment the patch number.
3. If a PR does not change the API but the output of some regression test changes, it should increment the patch number.

That all said, I don't feel strongly about any of this and am willing to defer to your preferred way of doing it. 😄

[msdemlei]

On Fri, Jul 07, 2023 at 05:34:54AM -0700, Vitaly wrote: What I'm concerned, now we increment only the third number (patch number) for those changes that include API changes. I guess we should increment minor number in this case. But if we decide to increment minor number on each API small change this number may become too big.

The big issue here are the upgrade scripts. On the one hand, these need to know the state in the database from the installed version (more or less; the increasing number of IF NOT EXISTS constructs in postgres help a bit to write more flexible upgrade scripts, but let's disregard that for the moment). This means that theoretically, each time we touch the SQL associated with pgsphere, we would need to bump the version number, and we will need a new upgrade script fragment. As evidenced by the current state of affairs, that's not pretty, and it does not scale particularly well.

One more question - should we increment minor or patch number each time when merging a new PR or introduce release plan?

Sooo, I'd argue against per-PR version numbers. Such a plan would increase the number of upgrade scripts linearly with the number of PRs merged, which, even if we keep them outside of the root directory in the future, sounds like a major nightmare, even if I don't expect *too* many API-changing PR against pgsphere. Hence, I'm all for a release plan. Versions are only "finalised" at release time, and that's when the upgrade scripts are frozen; before that the upgrade (previous) -> (current) is only preliminary. That means there's as many upgrade script fragments (and scripts in the package) as there are releases, which sounds manageable. That also means that we'll have to say: "If you build from a github checkout and upgrade a persistent database, you are on your own and will likely have tun run upgrade scripts into the final release manually." Now that I write this, I wonder how everyone else deals with this problem -- they must have it as well, no?

scripts. I guess we may implement some query function that returns the list of compiled features. It may help upgrade script to decide what to do.

I agree with that -- there needs to be some feature discovery with flexible APIs, not only for our scripts but for the clients. A jump-and-fall pattern (where you discover a feature by trying it and catch error conditions) in my experience is not well suited for SQL because of transactionality.

[vitcpp]

I like release plans too. We may consider to create a new release each 3 months. Each release will be tagged. What I would like to consider is what to do if a bug is found in the release. Should we create a branch from the tag and fix it in this branch? If yes, should we increment version number? I propose to fix releases with tags but to fix new bugs in the master only.

I also propose not to increment the version on each PR. I propose to increment it once when applying first PR after the release. Another condition when the release number should be incremented - if a new PR requires to increment minor or major numbers but before we incremented only patch number.

Upgrade scripts are created only to migrate from previous release to a new one. Intermediate PRs should just update existing upgrade scripts.

[vitcpp]

One more idea is to merge PR into a new branch master-dev. Once in 3 months we merge master-dev into master. This way, master branch will always contain the latest released version.

[msdemlei]

On Thu, Jul 13, 2023 at 06:13:07AM -0700, Vitaly wrote: I like release plans too. We may consider to create a new release each 3 months. Each release will be tagged. What I would like to

Given the pace of development in recent years, I'd say a timed schedule won't be needed; I'd say "there are new features and we believe they're mature enough" would be a fine criterion for when to release except that "we believe" and "mature enough" are hard to operationalise. I think optimally, we'd have a policy like "If a new piece of API is available in pgsphere, a pre-release is published that must be made operational at at least one installation; if no problems turn up, it is turned into a release after a month." Or so.

consider is what to do if a bug is found in the release. Should we create a branch from the tag and fix it in this branch? If yes, should we increment version number? I propose to fix releases with tags but to fix new bugs in the master only.

I'd say that depends on the severety of the bug. If it's security-critical, we probably should try to patch one or two previous releases for the benefit of distributors. Otherwise given the resources we have I don't think we can do backports, and upgrades ought to be easy enough for the deployers if they want fixes.

[esabol]

One more idea is to merge PR into a new branch master-dev. Once in 3 months we merge master-dev into master. This way, master branch will always contain the latest released version.

I'm not a fan of this model. I'd rather see release tagging, maybe even branching, and making tarballs available for download from https://github.com/postgrespro/pgsphere/releases.

[vitcpp]

I agree that dev branch is not the best solution. Once we may have intermediate changes in the master branch between assigned tags I would like to somehow notify developers that the version in master is the development version. I propose to add 'dev' suffix to version number in the master branch (like 1.2.2.dev) if it contains some intermediate commits but a new tag is not yet assigned. If a developer downloads the master branch (not a particular tag) it would be clear that the downloaded version contains intermediate changes.

When we decide to create a new tag we remote suffix 'dev' from the version number and assign the version tag. Next PR will increment version and add 'dev' suffix if it is an intermediate PR.

I agree it would be better to release a new version (create a new tag) by our decision, not to follow a predefined release plan.

With such approach we may still merge intermediate PRs without changing the version every time.

[vitcpp]

Another idea is to state that the tagged versions are the stable ones. The current version in master is not stable, its functionality or the version itself can be changed. Once we decide to create a new stable version just create a tag and increment the version in the master.

[esabol]

Another idea is to state that the tagged versions are the stable ones. The current version in master is not stable, its functionality or the version itself can be changed. Once we decide to create a new stable version just create a tag and increment the version in the master.

I like this idea the best, I think.

[vitcpp]

I think we have come to the agreement. Thank you very much for the participation. Give me please some time to prepare the versioning policy statement. I also plan to start assign version tags and create release artifacts.

[vitcpp]

I created a new 1.2.2 tag and I found that it appears in the tags list in the wrong order (https://github.com/postgrespro/pgsphere/tags). Furthermore, when I try to create a release artifact, it appears in the wrong order as well. The release notes for 1.2.3 should be changed in this case. I have to edit them. At the moment I have no ideas how to change the order of the tags. I appreciate if you share some ideas how to do it. Thank you in advance.

[vitcpp]

Dear All,

I created some historical tags: 1.1.5, 1.2.0, 1.2.1, 1.2.2. I looked for commits where PGSPHERE_VERSION in Makefile and default_version in pg_sphere.control were changed. There are some problems to find an appropriate commits for earlier tags than 1.1.5. I can't identify such commits.

I would appreciate if you verify that the assigned tags are properly specified.

Thank you in advance!

[esabol]

I created some historical tags: 1.1.5, 1.2.0, 1.2.1, 1.2.2.

Thanks!

[df7cb]

I'd suggest to call the tags good now, and close this.

And perhaps delete these branches in git:

• cleansing (merged)
• experimental (merged)
• pg14-compat (identical to merged commit)

[vitcpp]

I agree to remove the branches, cleansing and experimental branches are merged. pg14-compat changes seems to be merged in another complex PR. Lets remove these branches.

[pdowler]

I recently pulled the latest from master, got all the tags, and checked out specific versions (1.2.3, 1.3.1, 1.4.1) to build rpm packages (yum.postrgesql.org for PG itself). Aside from one small issue of my own making (PR for Makefile dist target) that went smoothly and I was able to create and servers with different versions to test.

So I would say I'm happy with the current tagging. 👍

[vitcpp]

As suggested, the following branches have been removed:

• cleansing
• experimental
• pg14-compat

[vitcpp]

Dear All,

I propose to close this Issue as completed. Do you have any objections?

[esabol]

No objection, @vitcpp.

[vitcpp]

Closed as completed.