Support serving platform extension catalogs in addition to non-platform extension catalogs

Question

Support serving platform extension catalogs in addition to non-platform extension catalogs

Closed this issue 2 years ago · 23 comments

At this point, the registry is used by the dev tools to pull in the list of the currently recommended platform releases and non-platform extension catalogs compatible with a given Quarkus core version. The platform extension catalogs are currently resolved as JSON artifacts from other Maven repos (such as Central or Red Hat). While this generally works well, there is a disadvantage of this approach when it comes to adjusting a part of the platform (extension) metadata, for example, for UI purposes. In this case, a new version of the platform will have to be released with the adjusted metadata even though code (or binary)-wise the new release would be identical to the previous one but still requiring QE and other procedures to be performed.

An alternative to that, would be to enable the registry to provide the platform extension catalogs too. The registry descriptor can already indicate whether a given registry provides platform extension catalogs or not, here is an example:

{
  "descriptor" : {
    "artifact" : "registry.quarkus.test:quarkus-registry-descriptor::json:1.0-SNAPSHOT"
  },
  "platforms" : {
    "artifact" : "registry.quarkus.test:quarkus-platforms::json:1.0-SNAPSHOT",
    "extension-catalogs-included" : true
  },

If extension-catalogs-included is true, the registry client will be resolving the platform extension catalogs from the registry instead of the other Maven repos. That would mean that the registry would have to be able to recognize Maven artifacts of the released platform descriptors such as https://repo1.maven.org/maven2/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor, https://repo1.maven.org/maven2/io/quarkus/platform/quarkus-camel-bom-quarkus-platform-descriptor and others.

And to be able to adjust some metadata, there would have to be an admin web/REST endpoint allowing to submit JSON metadata adjustments to the platform, category and/or extension metadata.

Answer 1 · 2022-04-13T17:18:41.000Z

@aloubyansky since this artifact is not a SNAPSHOT, the client needs to treat it as such given that it may change over time?

Answer 2 · 2022-04-13T18:16:16.000Z

No, the client will treat it as non-snapshot. We'll need some cache-refresh for these in code.quarkus and possibly other dev tools but it would be manually triggered by users, I think. I don't think we'll be changing the metadata very often.

Answer 3 · 2022-04-13T18:21:18.000Z

How about a new command in the CLI to purge the stale data? Something like quarkus registry cache clear

Answer 4 · 2022-04-13T19:12:32.000Z

We already have --refresh option in the CLI for registry cache refresh that is available in many commands. Perhaps we'll re-use it for this.

Answer 5 · 2022-04-20T12:04:07.000Z

During one of the discussions we considered an idea of making the registry provide not the final versions of the platform descriptors but snapshots, making the registry-provided platform descriptor overrides coordinates <platform-groupId>:<platform-artifactId>:<platform-version>:json:1.0-SNAPSHOT and enhancing the registry client to query registries that support platform metadata overrides.
The advantage of this approach is that the content of SNAPSHOT artifacts is meant to change, so it fits perfectly the use-case we are trying to support.
The disadvantage is that now the client will have to query remote registries for more artifacts (basically double the amount of platform artifacts to resolve).

Answer 6 · 2022-04-20T12:22:27.000Z

@aloubyansky I have implemented this idea in 6bd4434 (#92).
Do you think we should still allow artifacts using<platform-groupId>:<platform-artifactId>:<platform-version>:json:<platform-version>?

Answer 7 · 2022-04-20T12:33:20.000Z

I think I would keep these two options separate. I have test registries that provide the final versions of the descriptors and I want to keep that for them, because it's a lot easier to setup tests that way. But it doesn't mean that the registry.quarkus.io has to support this option. This is simply why I would like to keep the SNAPSHOT overrides and the final versions as separate registry options.
AFAIK, we don't have an actual requirement for it, atm. So, imo, you don't have to support it @gastaldi For the SNAPSHOT overrides though I'd need to enhance the registry client to support them.

Answer 8 · 2022-04-20T12:37:10.000Z

Sounds good, should we expose this endpoint only if QUARKUS_REGISTRY_PLATFORM_EXTENSION_CATALOG_INCLUDED=true or do you see any other use case that may use this endpoint when the extension-catalogs-included in the descriptor that is false?

Answer 9 · 2022-04-20T12:44:16.000Z

That's a good point. I don't think it's relevant if the platform overrides aren't enabled.

Answer 10 · 2022-04-22T16:26:28.000Z

Just found a bug in the ArtifactParser when dealing with classifiers.

quarkus-bom-quarkus-platform-descriptor-2.8.1.Final-1.0-SNAPSHOT.json wrongly resolves the classifier part to SNAPSHOT

Answer 11 · 2022-05-02T15:23:31.000Z

Enabling support for querying registries for platform overrides increases qs create time from ~1.3 sec to 8-9 sec.
Given that the overrides are currently not provided, it simply fails to resolve them on every project creation. But anyway, this looks like not a good idea.
As an alternative, we could probably simply query the registry for platoform-groupId:platform-artifactId:platform-version:json:1.0-SNAPSHOT and expect the complete catalog with the overrides already applied. And if that request fails we could fallback to other public Maven repos.

Answer 12 · 2022-05-02T15:24:52.000Z

Although until the registry provides those catalogs, it will still be taking 8-9 sec. But Once it starts to provide them, we won't need to resolve the platform catalogs from Central and MRRC.

Answer 13 · 2022-05-02T17:54:23.000Z

As an alternative, we could probably simply query the registry for platoform-groupId:platform-artifactId:platform-version:json:1.0-SNAPSHOT and expect the complete catalog with the overrides already applied. And if that request fails we could fallback to other public Maven repos.

Isn't that what #92 is doing, or am I missing something?

Answer 14 · 2022-05-02T18:06:43.000Z

I thought it was providing overrides and the main catalogs would still have to be resolved from Central, etc. So you are already providing complete catalogs as snapshots in that PR?

Answer 15 · 2022-05-02T18:08:50.000Z

@aloubyansky yes, it's the complete catalog with the overriden metadata applied

Answer 16 · 2022-05-02T18:09:32.000Z

Very good.

Answer 17 · 2022-05-02T18:13:08.000Z

BTW Have you had a chance to try it out yet?

Answer 18 · 2022-05-02T18:35:57.000Z

Haven't tried your PR yet but I did implement it in the registry client. I'll give it a try tomorrow.

Answer 19 · 2022-05-03T07:35:33.000Z

remind me again why not just releasing a new version of the JSON is not enough ?

Same burden on QE afaics wether we do that or add some of these tweaks ?

Answer 20 · 2022-05-03T10:12:49.000Z

Yes, a platform release process would be necessary. For today it is assumed a full QE is required.

Answer 21 · 2022-05-03T10:30:05.000Z

Okey so we have same QE burden whether we do a platform update (changing just overrides and have consistent metadata) than if we added some additional code into registry resolution that could morph the platform json in-place causing possible different metadata dependent on user sequence and access to internet?

so why do custom overrides and not just avoid the whole issue?

Answer 22 · 2022-05-03T11:17:13.000Z

It's not that bad. It's a reasonable alternative to have.

Answer 23 · 2022-05-03T11:53:58.000Z

@gastaldi doesn't seem to work, unless I'm missing something

[aloubyansky@localhost playground]$ qs create --verbose
Creating an app (default project type, see --help).
[DEBUG] Creating a new project with initial parameters: CreateApp{gav=TargetGAVGroup [groupId=org.acme, artifactId=code-with-quarkus, version=1.0.0-SNAPSHOT, gav=null], quarkusVersion=TargetQuarkusVersionGroup{stream=null, platformBom=null}, targetBuildTool=TargetBuildToolGroup [gradle=false, gradleKotlinDsl=false, jbang=false, maven=false], targetLanguage=TargetLanguageGroup [java=<<detect java runtime version>>, kotlin=false, scala=false, sourceType=null], codeGeneration=CodeGenerationGroup [includeCode=true, includeWrapper=true, packageName=null], extensions=[], project=BaseCreateCommand [outputPath=null, registryClientRegistryClientMixin [useRegistryClient=true], projectDirName=null, projectRootPath=null, targetDirectory=null, values={}], properties={}}
[DEBUG] Creating an app using the following settings: {project_artifactId=code-with-quarkus, app-config={}, extensions=[], quarkus.create-project.no-buildtool-wrapper=false, project_groupId=org.acme, quarkus.create-project.no-code=false, project_version=1.0.0-SNAPSHOT, source_type=JAVA, java_target=11}
[DEBUG] Resolving Quarkus extension catalog for TargetQuarkusVersionGroup{stream=null, platformBom=null}
[DEBUG] Resolved registry descriptor io.quarkus.registry:quarkus-registry-descriptor:json:1.0-SNAPSHOT from registry.quarkus.io
[DEBUG] Resolving platform catalog io.quarkus.registry:quarkus-platforms:json:1.0-SNAPSHOT
[DEBUG] Resolving platform extension catalog io.quarkus.platform:quarkus-bom-quarkus-platform-descriptor:json:2.8.2.Final:1.0-SNAPSHOT
Looking for the newly published extensions in registry.quarkus.io
[ERROR] ❗  Unable to create project: Failed to resolve extension catalog of io.quarkus.platform:quarkus-bom:pom:2.8.2.Final from Maven repository registry.quarkus.io (http://localhost:8080/maven)

The snapshots for platform catalogs are enabled:

{
  "descriptor" : {
    "artifact" : "io.quarkus.registry:quarkus-registry-descriptor::json:1.0-SNAPSHOT"
  },
  "platforms" : {
    "artifact" : "io.quarkus.registry:quarkus-platforms::json:1.0-SNAPSHOT",
    "extension-catalogs-included" : true
...