Allow wildcards and ranges in the `runPluginVerifier` > `ideVersions` list

Question

Allow wildcards and ranges in the `runPluginVerifier` > `ideVersions` list

philCryoport opened this issue 9 months ago · 14 comments

Describe the need of your request

Unless there's something I haven't figured out by RTFMing the instructions, we need to list

{
"each",
"and",
"every",
"specific",
"IDE",
"version",
"we",
"want",
"the",
"runPluginIdentifier",
"to",
"check"
}

In the case of the jasyncapi AsyncAPI Preview plugin, our under-review version has the following code block:

tasks.getByName<org.jetbrains.intellij.tasks.RunPluginVerifierTask>("runPluginVerifier") {
    ideVersions.set(listOf(
        "2022.3",
        "2022.3.1",
        "2022.3.2",
        "2022.3.3",
        "2023.1",
        "2023.1.1",
        "2023.1.2",
        "2023.1.3",
        "2023.1.4",
        "2023.1.5",
        "2023.2",
        "2023.2.1",
        "2023.2.2",
        "2023.2.3",
        "2023.2.4",
        "2023.2.5",
        "2023.3",
        "2023.3.1",
        "2023.3.2"
}

It would be nice instead to use ranges.

Proposed solution

It would be nice instead to use ranges. For instance, if we want to verify GA IDE versions ranging from the first 2022 version through the latest 2023 version, inclusive, it would be great if we could just write something along the lines of:

{
"rangeStart" : "2022.*",
"rangeEnd" : "2023.*",
"releaseType" : "GA"
}

...or:
{
"GA2022.",
"GA2023.",
}

Alternatives you've considered

As mentioned above, RTFM'd and didn't see how to use wildcards or ranges.

Additional context

N/A

Answer 1 · 2024-04-03T13:08:14.000Z

If no specific IDEs/versions are listed, it will verify against given since/until-build by default. Does that match your use case?

Answer 2 · 2024-04-22T19:59:47.000Z

@YannCebron sinceBuild and since not working for my config file's runPluginVerifier section:

Code change:

I removed the list of specific versions in runPluginVerifier -- and instead tried:

ideVersions.set(sinceBuild("2022.3")

Threw an error:

2: Task failed with an exception.
-----------
* Where:
Build file '/Users/pschlesinger/GitHub/asyncapi/jasyncapi-idea-plugin/build.gradle.kts' line: 100

* What went wrong:
Script compilation error:

  Line 100:     ideVersions.set(sinceBuild("2022.3"))
                                ^ Unresolved reference: sinceBuild

The stacktrace only lists org.gradle.* -- nothing references jetbrains or intellij.

I also tried ideVersions.set(since("2022.3") and ideVersions.set(since("IU-223")

Same error.

Tried to RTFM to find the syntax. It wants a list:

The options link states:

I dug into the Github repo, searching for ideVersions. Here too it is expecting a list.

FYI here's what happens if I leave the `ideVersions.set()` call completely out of the `runPluginVerifier` section

> Task :runPluginVerifier
Starting the IntelliJ Plugin Verifier 1.365
Verification reports directory: /foo/GitHub/asyncapi/jasyncapi-idea-plugin/build/reports/pluginVerifier
2024-04-22T12:30:06 [main] INFO  verification - Reading IDE /foo/.cache/pluginVerifier/ides/IC-241.15989.69
2024-04-22T12:30:06 [main] INFO  c.j.p.options.OptionsParser - Reading IDE from /foo/.cache/pluginVerifier/ides/IC-241.15989.69
2024-04-22T12:30:06 [main] INFO  c.j.p.options.OptionsParser - Using Java runtime from /foo/.gradle/caches/modules-2/files-2.1/com.jetbrains/jbre/jbr_jcef-17.0.10-osx-aarch64-b1207.12/extracted/jbr_jcef-17.0.10-osx-aarch64-b1207.12/Contents/Home
2024-04-22T12:30:12 [main] INFO  verification - Reading IDE /foo/.cache/pluginVerifier/ides/IC-2023.3.6
2024-04-22T12:30:12 [main] INFO  c.j.p.options.OptionsParser - Reading IDE from /foo/.cache/pluginVerifier/ides/IC-2023.3.6
2024-04-22T12:30:12 [main] INFO  c.j.p.options.OptionsParser - Using Java runtime from /foo/.gradle/caches/modules-2/files-2.1/com.jetbrains/jbre/jbr_jcef-17.0.10-osx-aarch64-b1207.12/extracted/jbr_jcef-17.0.10-osx-aarch64-b1207.12/Contents/Home
2024-04-22T12:30:16 [main] INFO  verification - Reading IDE /foo/.cache/pluginVerifier/ides/IC-2023.2.6
2024-04-22T12:30:16 [main] INFO  c.j.p.options.OptionsParser - Reading IDE from /foo/.cache/pluginVerifier/ides/IC-2023.2.6
2024-04-22T12:30:16 [main] INFO  c.j.p.options.OptionsParser - Using Java runtime from /foo/.gradle/caches/modules-2/files-2.1/com.jetbrains/jbre/jbr_jcef-17.0.10-osx-aarch64-b1207.12/extracted/jbr_jcef-17.0.10-osx-aarch64-b1207.12/Contents/Home
2024-04-22T12:30:20 [main] INFO  verification - Reading IDE /foo/.cache/pluginVerifier/ides/IC-2023.1.6
2024-04-22T12:30:20 [main] INFO  c.j.p.options.OptionsParser - Reading IDE from /foo/.cache/pluginVerifier/ides/IC-2023.1.6
2024-04-22T12:30:20 [main] INFO  c.j.p.options.OptionsParser - Using Java runtime from /foo/.gradle/caches/modules-2/files-2.1/com.jetbrains/jbre/jbr_jcef-17.0.10-osx-aarch64-b1207.12/extracted/jbr_jcef-17.0.10-osx-aarch64-b1207.12/Contents/Home
2024-04-22T12:30:24 [main] INFO  verification - Reading IDE /foo/.cache/pluginVerifier/ides/IC-223.8836.26
2024-04-22T12:30:24 [main] INFO  c.j.p.options.OptionsParser - Reading IDE from /foo/.cache/pluginVerifier/ides/IC-223.8836.26
2024-04-22T12:30:24 [main] INFO  c.j.p.options.OptionsParser - Using Java runtime from /foo/.gradle/caches/modules-2/files-2.1/com.jetbrains/jbre/jbr_jcef-17.0.10-osx-aarch64-b1207.12/extracted/jbr_jcef-17.0.10-osx-aarch64-b1207.12/Contents/Home
2024-04-22T12:30:27 [main] INFO  verification - Reading plugin to check from /foo/GitHub/asyncapi/jasyncapi-idea-plugin/build/distributions/jasyncapi-idea-plugin-2.5.0+jre17.zip
2024-04-22T12:30:31 [main] INFO  verification - Task check-plugin parameters:
Scheduled verifications (5):
com.asyncapi.plugin.idea:2.5.0+jre17 against IC-241.15989.69, com.asyncapi.plugin.idea:2.5.0+jre17 against IC-233.15026.9, com.asyncapi.plugin.idea:2.5.0+jre17 against IC-232.10300.40, com.asyncapi.plugin.idea:2.5.0+jre17 against IC-231.9414.13, com.asyncapi.plugin.idea:2.5.0+jre17 against IC-223.8836.26

2024-04-22T12:30:32 [main] INFO  verification - Finished 1 of 5 verifications (in 1.3 s): IC-223.8836.26 against com.asyncapi.plugin.idea:2.5.0+jre17: Compatible. 2 override-only API usage violations
2024-04-22T12:30:32 [main] INFO  verification - Finished 2 of 5 verifications (in 1.3 s): IC-233.15026.9 against com.asyncapi.plugin.idea:2.5.0+jre17: Compatible. 2 override-only API usage violations
2024-04-22T12:30:33 [main] INFO  verification - Finished 3 of 5 verifications (in 1.4 s): IC-232.10300.40 against com.asyncapi.plugin.idea:2.5.0+jre17: Compatible. 2 override-only API usage violations
2024-04-22T12:30:33 [main] INFO  verification - Finished 4 of 5 verifications (in 1.4 s): IC-241.15989.69 against com.asyncapi.plugin.idea:2.5.0+jre17: Compatible. 2 override-only API usage violations
2024-04-22T12:30:33 [main] INFO  verification - Finished 5 of 5 verifications (in 1.4 s): IC-231.9414.13 against com.asyncapi.plugin.idea:2.5.0+jre17: Compatible. 2 override-only API usage violations
Plugin com.asyncapi.plugin.idea:2.5.0+jre17 against IC-223.8836.26: Compatible. 2 override-only API usage violations
Override-only API usages (2):
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.5.0+jre17 against IC-233.15026.9: Compatible. 2 override-only API usage violations
Override-only API usages (2):
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.5.0+jre17 against IC-232.10300.40: Compatible. 2 override-only API usage violations
Override-only API usages (2):
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.5.0+jre17 against IC-241.15989.69: Compatible. 2 override-only API usage violations
Override-only API usages (2):
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

Plugin com.asyncapi.plugin.idea:2.5.0+jre17 against IC-231.9414.13: Compatible. 2 override-only API usage violations
Override-only API usages (2):
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v2.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
    #Invocation of override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName()
        Override-only method com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() is invoked in com.asyncapi.plugin.idea.extensions.index.v3.AsyncAPISpecificationIndex.getName() : IndexId. This method is marked with @org.jetbrains.annotations.ApiStatus.OverrideOnly annotation, which indicates that the method must be only overridden but not invoked by client code. See documentation of the @ApiStatus.OverrideOnly for more info.
Dynamic Plugin Eligibility:
    Plugin can probably be enabled or disabled without IDE restart

2024-04-22T12:30:33 [main] INFO  verification - Total time spent downloading plugins and their dependencies: 0 ms
2024-04-22T12:30:33 [main] INFO  verification - Total amount of plugins and dependencies downloaded: 0 B
2024-04-22T12:30:33 [main] INFO  verification - Total amount of space used for plugins and dependencies: 0 B
2024-04-22T12:30:33 [main] INFO  verification - Verification reports for com.asyncapi.plugin.idea:2.5.0+jre17 saved to /foo/GitHub/asyncapi/jasyncapi-idea-plugin/build/reports/pluginVerifier/IC-223.8836.26
2024-04-22T12:30:33 [main] INFO  verification - Verification reports for com.asyncapi.plugin.idea:2.5.0+jre17 saved to /foo/GitHub/asyncapi/jasyncapi-idea-plugin/build/reports/pluginVerifier/IC-233.15026.9
2024-04-22T12:30:33 [main] INFO  verification - Verification reports for com.asyncapi.plugin.idea:2.5.0+jre17 saved to /foo/GitHub/asyncapi/jasyncapi-idea-plugin/build/reports/pluginVerifier/IC-232.10300.40
2024-04-22T12:30:33 [main] INFO  verification - Verification reports for com.asyncapi.plugin.idea:2.5.0+jre17 saved to /foo/GitHub/asyncapi/jasyncapi-idea-plugin/build/reports/pluginVerifier/IC-241.15989.69
2024-04-22T12:30:33 [main] INFO  verification - Verification reports for com.asyncapi.plugin.idea:2.5.0+jre17 saved to /foo/GitHub/asyncapi/jasyncapi-idea-plugin/build/reports/pluginVerifier/IC-231.9414.13
2024-04-22T12:30:33 [main] INFO  verification - Total time spent in plugin verification: 26 s 966 ms

What is the correct syntax to specify for the runPluginVerifier that it verify the plugin-under-development against IntellJ releases from 2022.3 onward?

Answer 3 · 2024-05-07T18:20:07.000Z

Hi @hsz and @YannCebron ,

Hope this follow-up is ok. When you get a moment, would you please take a look at and respond to my comment-reply above? Thanks!

Answer 4 · 2024-05-13T06:53:00.000Z

Hi,
when running the Plugin Verifier, since-build and until-build are taken from your plugin descriptor. There is no need to specify that in the Gradle task explicitly again.

When your plugin has since-build value of 223, it indicates compatibility with 2022.3 and later. This will be used by the Plugin Verifier, too. See Build Number Ranges in the IntelliJ Platform Plugin SDK.

Answer 5 · 2024-05-13T20:16:54.000Z

Hi, when running the Plugin Verifier, since-build and until-build are taken from your plugin descriptor. There is no need to specify that in the Gradle task explicitly again.

When your plugin has since-build value of 223, it indicates compatibility with 2022.3 and later. This will be used by the Plugin Verifier, too. See Build Number Ranges in the IntelliJ Platform Plugin SDK.

Hi @novotnyr

Somehow sinceBuild() and untilBuild() are not being fully used for runPluginVerifier in our build.gradle.kts file.

Let's start with what we have now: a laundry list of every ideVersion we'd like to have runPluginVerifier to verify:

tasks.getByName<org.jetbrains.intellij.tasks.PatchPluginXmlTask>("patchPluginXml") {
    sinceBuild.set("223")
    untilBuild.set("241.*")
}

tasks.getByName<org.jetbrains.intellij.tasks.RunPluginVerifierTask>("runPluginVerifier") {
	verifierVersion.set("1.365")
    ideVersions.set(listOf(
        "2022.3",
        "2022.3.1",
        "2022.3.2",
        "2022.3.3",
        "2023.1",
        "2023.1.1",
        "2023.1.2",
        "2023.1.3",
        "2023.1.4",
        "2023.1.5",
        "2023.2",
        "2023.2.1",
        "2023.2.2",
        "2023.2.3",
        "2023.2.4",
        "2023.2.5",
        "2023.3",
        "2023.3.1",
        "2023.3.2",
        "2023.3.3",
        "2023.3.4",
        "2023.3.5",
        "2023.3.6",
        "2024.1"
    ))
}

I captured the output of ./gradlew :runPluginVerifier:
with_entire_listOf_ideVersions.log

Here, if I want an IDE version runPluginVerifier verified, I have to specify it. Kinda dumb that I need to manually curate that. I should be able to just specify a range of versions, right?

If I read your above statement correctly, I should able to just rely on:

tasks.getByName<org.jetbrains.intellij.tasks.PatchPluginXmlTask>("patchPluginXml") {
    sinceBuild.set("223")
    untilBuild.set("241.*")
}

...and I should be able to remove the ideVersions.set(listOf(...)), CORRECT?

So I did that. I commented out the entire ideVersions.set(listOf(...)) statement. Result from ./gradlew :runPluginVerifier:
removed_entire_listOf_ideVersions.log

DIFFERENCE: a ton of verifications that DID NOT OCCUR.

FULL DIFF:
compare_withListOfIdeVersions_versus_noListOfIdeVersions.txt

To my eyes, sinceBuild() and untilBuild() are NOT being interpreted correctly.

I'm hoping that maybe I'm missing something...

HELP?

Answer 6 · 2024-05-13T20:44:54.000Z

At least in 2.0, that does not work @novotnyr

<idea-plugin xmlns:xi="http://www.w3.org/2001/XInclude">
  <version>1.4.3</version>
  <idea-version since-build="232" />

yields the following:

[org.jetbrains.intellij.platform] Filtering releases with since='232.8660', until='232.99999', types='IC', channels='EAP,MILESTONE,BETA,RELEASE,CANARY,PATCH,RC,PREVIEW'

The until build isn't going up to max like it does on the marketplace

Answer 7 · 2024-05-13T20:48:23.000Z

@abrooksv wrote:

At least in 2.0, that does not work

Apologies, I'm holding off on gradle-intelli-plugin v2.0 until it goes production...

Answer 8 · 2024-05-14T14:33:15.000Z

Let's focus on the 1.x now.

Plugin Verifier is behaving correctly. However, when you leave out the since-build and until-build, you'll get the verification against most recent builds in each Platform version.

E. g. I have a demo plugin that states in the plugin descriptor:

since-build: 223
until-build: 241.*

These values are used in the patchPluginXml. Then, I will get the following verification targets:

IC-223.8836.26
IC-231.9414.13
IC-232.10300.40
IC-233.15026.9
IC-241.17011.2

@philCryoport, from your description, I assume you would like to verify against all point releases, right?

@hsz please confirm the behavior of the Gradle IntelliJ Plugin 1.x.

Answer 9 · 2024-05-14T19:27:22.000Z

Let's focus on the 1.x now.

👍

Plugin Verifier is behaving correctly. However, when you leave out the since-build and until-build, you'll get the verification against most recent builds in each Platform version.

E. g. I have a demo plugin that states in the plugin descriptor:
* since-build: 223

* until-build: 241.*
These values are used in the patchPluginXml. Then, I will get the following verification targets:
* IC-223.8836.26

* IC-231.9414.13

* IC-232.10300.40

* IC-233.15026.9

* IC-241.17011.2
@philCryoport, from your description, I assume you would like to verify against all point releases, right?

Yes please!

2022.3 and its patch releases
2023.x.y -- major, minor, and patch releases
2024.x.y -- major, minor, and patch releases

Answer 10 · 2024-05-15T06:42:55.000Z

@hsz please note that even the listProductReleases is acting in a non-obvious way.

    listProductsReleases {
        sinceVersion = "2023.3.1"
        untilVersion = "2023.3.6"
    }

Emits:

IC-2023.3.6

I would expect that it should list all point releases between .1 and .6.

Answer 11 · 2024-06-20T23:39:29.000Z

Following up with @hsz / @YannCebron : any thoughts on @novotnyr 's comment above that "even the listProductReleases is acting in a non-obvious way"?

Answer 12 · 2024-06-27T15:04:54.000Z

AFAIU the explanation is that intermediate bugfix updates are no longer relevant, thus only the latest bugfix release is included (assuming all users update to latest bugfix updates always).

Answer 13 · 2024-06-28T22:08:11.000Z

AFAIU the explanation is that intermediate bugfix updates are no longer relevant, thus only the latest bugfix release is included (assuming all users update to latest bugfix updates always).

Ok I can kind of understand that

Answer 14 · 2024-07-01T15:12:25.000Z

Can we close this issue then?