OpenSearch: Automate increment to next development iteration

Question

OpenSearch: Automate increment to next development iteration

dblock opened this issue 3 years ago · 28 comments

Answer 1 · 2021-12-17T20:35:09.000Z

@dblock what do you think about updating the component release template with these as steps? Then we could create a campaign to automation this step in each repository, which a base case that hits 80% of them with a copypasta workflow

Answer 2 · 2022-03-23T17:11:12.000Z

@dblock is there any further blocker for other component teams to set this up themselves? Should this be added as a campaign for 2.0.0?

Answer 3 · 2022-03-23T17:32:04.000Z

It didn't work for common-utils: https://github.com/opensearch-project/common-utils/actions/runs/2000270385, needs to be debugged, opensearch-project/common-utils#138.

It didn't work for OpenSearch, https://github.com/opensearch-project/OpenSearch/actions/runs/2000274118, the PR GHA needed to be added to permissions (I did that).

It didn't quite work for job-scheduler, opensearch-project/job-scheduler#148, DCO was one problem.

Before adding more we need to fix these, but I didn't get to it. If someone wants to pick these up, go for it!

Answer 4 · 2022-03-23T17:43:29.000Z

@dblock I added the secrets for common-utils. I will fix DCO for both job-scheduler and common-utils.

Answer 5 · 2022-03-23T19:55:55.000Z

We also need this in https://github.com/opensearch-project/OpenSearch/blob/main/.github/workflows/version.yml#L56, and possibly in other places? Would you mind adding it too please?

Answer 6 · 2022-03-23T20:29:18.000Z

We also need this in https://github.com/opensearch-project/OpenSearch/blob/main/.github/workflows/version.yml#L56, and possibly in other places? Would you mind adding it too please?

Yup added it in OpenSearch version.yml PR opensearch-project/OpenSearch#2572. I will see if there are any other pull request workflows that need the signoff option.

Answer 7 · 2022-06-18T14:43:00.000Z

Proposed Solution:

Automate this version increment process across all plugin components, that increment’s the version, modifies the required files and raise a PR that awaits for plugin team approval, this way an engineer need not spend hours of time to increment the version for all plugin components.

Solution Details:

The following details uses sql component, version increment process, to demonstrate the solution workflow.

Each plugin will have a gradle property opensearch.version stored in gradle.properties file.
Example opensearch.version=2.0.0-SNAPSHOT https://github.com/prudhvigodithi/sql/blob/main/gradle.properties

With current/existing setup depending on OpenSearch version, the plugins version is inferred, this property is injected using gradle system properties.
Moving forward this version will be inferred from plugin specific gradle property and rest of the flow just depend upon this opensearch.version gradle property.

Note: This can be debated if opensearch.version should be passed from CI during distribution runtime or straight away use from plugin own gradle properties.
Example:
Existing flow -Dopensearch.version=$VERSION from build script is inferred from manifest
Proposed Flow, to directly use project properties.
Each Plugin, gradle build system will have two additional tasks which are responsible to increment the version and modify the required files.
1. setVersion : This task will update the gradle property opensearch.version and updates the value in gradle.properties file.
2. versionIncrement : This task is configurable and depends on the plugin requirement to modify any files where version is present and that needs to incremented, this task uses ant.replaceregexp to parse the required files that are added in include() section of the task. Example for sql
For JSON files to increment the version, plugin teams can even include gradle node processes inside versionIncrement task, instead of RegEx parsing. However in the sql example i have used RegEx parsing to increment the version.
https://ant.apache.org/manual/Tasks/replaceregexp.html
The task setVersion is dependent on versionIncrement task, so when called ./gradlew setVersion -PnewVersion=2.1.0-SNAPSHOT (this task will be called using GitHub workflow explained below) the gradle property in gradle.propperties file gets updated to opensearch.version=2.1.0-SNAPSHOT and all include() section files will be replaced from 2.0.0 to 2.1.0 or 2.1.0.0 depending upon the value in the file.
Using GitHub workflows the plugin specific branch is compared against the OpenSearch specific branch to check for OpenSearch version, if it’s different, i.e if OpenSearch is incremented and not the plugin, the workflow will auto raise a PR.
Example:
plugin 1.3 branch will look for OpenSearch repo 1.3 branch, checks if it has same version.
plugin 2.x branch will look for OpenSearch repo 2.x branch, checks if it has same version.
If the OpenSearch version is different (condition check done using github workflow) from what the plugin has the workflow will execute ./gradlew setVersion -PnewVersion=<OpenSearch_Version> which performs the actual version increment process and raises a PR.
Sample PR’s: prudhvigodithi/sql#3 prudhvigodithi/sql#2

Note: This workflow can be scheduled using cron, on after release tag cut. This workflow can be part of each plugin .github/workflows/ or can be researched to be part of central build repo and target to create PR’s across all plugin components.

Answer 8 · 2022-06-20T19:46:14.000Z

Hey please add your thoughts for the above added solution.
@dblock, @bbarani, @peternied, @VachaShah, @ohltyler

Answer 9 · 2022-06-21T19:19:41.000Z

I like it! Make it work.

Answer 10 · 2022-06-24T15:36:23.000Z

For non-Gradle projects:

Since Gradle support seamless integration of tasks for build automation, it’s preferable to add a gradle project to existing repos, for this integration a repo needs build.gradle, and gradle installation files. Then the implementation flow would be same as specified above to add setVersion and versionIncrement task.

The following details uses alerting-dashboards-plugin component, version increment process, to demonstrate the solution workflow.
Existing Sample version increment PR for alerting-dashboards-plugin
opensearch-project/alerting-dashboards-plugin#277

Sample build.gradle file

/*
 * Copyright OpenSearch Contributors
 * SPDX-License-Identifier: Apache-2.0
 */

buildscript {
    ext {
        opensearch_version = project.property('opensearch.version')
    }
}

task versionIncrement {
    final String workingDir = project.buildDir.toString() + "/../"
    if(project.hasProperty('newVersion')) {
         println 'Set Project to new Version '+newVersion.tokenize('-')[0]
        ant.replaceregexp(match: opensearch_version.tokenize('-')[0], replace: newVersion.tokenize('-')[0], flags:'g', byline:true) {
            fileset(dir: workingDir) {
                include(name: ".github/workflows/cypress-workflow.yml")
            }
        }
        ant.replaceregexp(match:'"version": "\\d+.\\d+.\\d+.\\d+', replace:'"version": ' + '"' + newVersion.tokenize('-')[0] + '.0', flags:'g', byline:true) {
            fileset(dir: workingDir) {
                include(name: "package.json")
                include(name: "opensearch_dashboards.json")
            }
        }
        ant.replaceregexp(file:'opensearch_dashboards.json', match:'"opensearchDashboardsVersion": "\\d+.\\d+.\\d+', replace:'"opensearchDashboardsVersion": ' + '"' + newVersion.tokenize('-')[0], flags:'g', byline:true)
    }
}
task setVersion(dependsOn: versionIncrement) {
    if(project.hasProperty('newVersion')) {
        ant.propertyfile(
            file: "gradle.properties") {
            entry( key: "opensearch.version", value: "${newVersion}")
        }
    }
}

Existing version:

yarn version

info Current version: *2.0.1.0*

Executing the gradle task:
./gradlew setVersion -PnewVersion=2.1.0-SNAPSHOT

Version Increment:

yarn version

Current version: *2.1.0.0*

The alternative approach is to use npm-version, but for each repo the version is present in more than one file, this requires additional linux parsing logic apart from regular npm commands.
Going with gradle way ensures consistent solution which is in par with other gradle projects.

thoughts @dblock @bbarani @peterzhuamazon ?

Answer 11 · 2022-06-24T16:50:20.000Z

Since most operations are ant.relaceregex, and I think all should be basically identical or at least very similar, I think you can go one step further and implement an extension in which a plugin only has to list the files which contain a version number, and move all the logic into the shared plugin implementation, e.g.:

incrementVersion {
   fileset(dir: workingDir) {
       include(name: ".github/workflows/cypress-workflow.yml")
       include(name: 'build.gradle')
   }
}

Answer 12 · 2022-06-24T18:59:48.000Z

Hey @kavilla @tmarkley can you please add your thoughts this proposed solution?
#1375 (comment)

Answer 13 · 2022-06-27T15:36:05.000Z

How about we exclude fileset(dir: workingDir) as the default directory is the build directory, and following as plugin extension?

incrementVersion {
     match: "<PATTERN>"
     replace: "<PATTERN>"
     files = [".github/workflows/cypress-workflow.yml", "build.gradle"]
}

Above should work for all regular files.
Sometimes for JSON files, there needs to be an exact match pattern passed as shown

ant.replaceregexp(file:'opensearch_dashboards.json', match:'"opensearchDashboardsVersion": "\\d+.\\d+.\\d+', replace:'"opensearchDashboardsVersion": ' + '"' + newVersion.tokenize('-')[0], flags:'g', byline:true)

In json sometimes the project version matches with the version of other packages, example

{
 "version": "2.0.0.0",
 "devDependencies": { 
     "ts-node": "^2.0.0"
   }
}

in this case we need an explicit match pattern just to replace the "version": "2.0.0.0", and not "ts-node": "^2.0.0"

        ant.replaceregexp(match:'"version": "\\d+.\\d+.\\d+.\\d+', replace:'"version": ' + '"' + newVersion.tokenize('-')[0] + '.0', flags:'g', byline:true) {
            fileset(dir: workingDir) {
                include(name: "package.json")
            }
        }

In this case we might need the extension something as

incrementVersion {
  regularFiles {
     match = "<PATTERN>"
     replace = "<PATTERN>"
     files = [".github/workflows/cypress-workflow.yml", "build.gradle"]
  }
  jsonFiles {
     match = "<PATTERN>"
     replace = "<PATTERN>"
     files = ["package.json"]
  }
}

(or)

incrementVersion {
     regularMatch = "<PATTERN>"
     regularReplace = "<PATTERN>"
     jsonMatch = "<PATTERN>"
     jsonReplace = "<PATTERN>"
     regularFiles = [".github/workflows/cypress-workflow.yml", "build.gradle"]
     jsonFiles = ["package.json"]
}

Thoughts @dblock @bbarani ?

Answer 14 · 2022-06-28T15:53:52.000Z

Since the version increment logic is different per plugin, each having multiple files to update the version (not a common logic).

Example:
opensearch-project/performance-analyzer-rca#189
opensearch-project/common-utils#180
opensearch-project/alerting#464
opensearch-project/asynchronous-search#149
opensearch-project/index-management#374

Considering gradle plugin will be tightly coupled with the parsing logic which will add some restrictions, so in this case its flexible to go with individual plugin task in which the parsing logic to increment the version present can be accommodated with plugins scenario.

@dblock @dblock

Answer 15 · 2022-06-28T18:31:25.000Z

@prudhvigodithi If there's no common logic then there's no need for a plugin, a task with the same name can be implemented in every plugin by "convention".

Answer 16 · 2022-07-05T20:52:14.000Z

@prudhvigodithi If there's no common logic then there's no need for a plugin, a task with the same name can be implemented in every plugin by "convention".

I like this approach since Dashboards and other Node based packages for example dont use Gradle for the build system and migrating away from it would be a huge lift (especially for a version upgrade). Instead a build script convention/contract allows each repo to use the existing build system but in a way that makes building and releasing all repos and plugins easier.

Answer 17 · 2022-07-05T21:12:11.000Z

Hey @ashwin-pc

Instead a build script convention/contract allows each repo to use the existing build system but in a way that makes building and releasing all repos and plugins easier.

Could you please elaborate your thoughts for dashboard plugins that are not gradle?
The idea here is for not only related to version upgrade (in this case there is be multiple scripts for each purpose) but also should support future requirements (like a plug and play model :) ).

Answer 18 · 2022-07-12T01:24:49.000Z

Hey @ashwin-pc

Instead a build script convention/contract allows each repo to use the existing build system but in a way that makes building and releasing all repos and plugins easier.

Could you please elaborate your thoughts for dashboard plugins that are not gradle? The idea here is for not only related to version upgrade (in this case there is be multiple scripts for each purpose) but also should support future requirements (like a plug and play model :) ).

sure, Dashboards and its plugins have their own bundler webpack that is used for all automations including to transform and build the plugin. These tasks are run using nodejs scripts. For simple tasks we just use node scripts themself and for more complex transformations and automations we use a combination of custom webpack plugins and node scripts. I'm not sure of all the future requirements that we intend to support in the build process, but i'm pretty sure that it would be fairly straightforward to implement these automations using the existing tools within the repo.

Answer 19 · 2022-07-12T22:56:38.000Z

Renaming to task updateVersion from [versionIncrement](#1375 (comment) following) opensearch-project/opensearch-plugin-template-java#32.

Answer 20 · 2022-09-08T20:16:57.000Z

This META can be closed, the version increment for OpenSearch plugins is now automated with a single click using the GH workflow (Sample Run: https://github.com/opensearch-project/opensearch-build/actions/runs/2967596360)
(The workflow only fails if the release branch does not exists from the plugin end, then it has to be re-triggered once the branch is created by the plugin team.)

The only pending issue open from this META are:
The branch sync, since this is an ongoing process in discussion with plugins, the version increment workflow will continue to compare against Core to raise an auto version increment PR.
@dblock @bbarani

Answer 21 · 2022-09-08T21:48:15.000Z

But so it's not automated until it is, let's reopen it :) ~~Which plugins don't follow the branching strategy that keep breaking this?~~ It's a long list in opensearch-project/opensearch-plugins#142?

Answer 22 · 2022-09-15T03:11:16.000Z

The automation to add the branch entry to the version increment workflow is now done using the existing manifest workflow
Related Issue: #2601
Related PR: #2602

Answer 23 · 2022-09-15T16:28:43.000Z

The version increment workflow, will be now triggered daily with cron expression cron: 0 0 * * * added to the github workflow
Related PR: #2623

Answer 24 · 2022-09-16T14:33:48.000Z

Hey @dblock the automation is now in place, the branch entry is added via the manifest workflow, sample PR: https://github.com/opensearch-project/opensearch-build/pull/2628/files, and the workflow is now triggered using cron: 0 0 * * *, if you have any thoughts please let me know, else I will proceed to close this.
Thank you

Answer 25 · 2022-09-16T15:35:26.000Z

Thanks, closing this issue, please feel free to re-open if required.

Answer 26 · 2022-09-16T16:42:00.000Z

@prudhvigodithi With Also handle BWC tests as part of version increment automation #2373 this process isn't fully automated. Are we tracking that issue separately?

Answer 27 · 2022-09-16T17:13:51.000Z

Hey @peternied as mentioned, the version increment process is generic and is not related to any BWC tests, each plugin right now follow their own way to add the BWC test version change and also the version change files are not common, this has to be handled from plugin team and this change is specific with a major version release.
Thank you
@bbarani @zelinh

Answer 28 · 2022-09-16T19:21:37.000Z

@peternied this can be extended within the plugin own updateVersion (example for security) gradle task, so when called using version-increment workflow, the BWC test version will also be incremented.

OpenSearch: Automate increment to next development iteration

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Acceptance Criteria

Additional Acceptance Criteria

Campaign Issues:

OpenSearch

OpenSearch Dashboards

Remove the dependency with hardcoded zips for 1.x versions:

Describe alternatives you've considered

Additional context

Proposed Solution:

Solution Details: