renovatebot/azure-devops-marketplace

Add additional metadata

jessehouwing opened this issue · 17 comments

jessehouwing commented

Currently the metadata only contains the versions for each task, but it would be relatively simple to add:

  • Preview (isStable: false)
  • Deprecated
  • Repository.Uri (SourceUrl)
  • Marketplace Uri (Homepage)

Since all this data is available.

This would look something like this:

@JamieMagee @viceice What do you think and am I correct in guessing this would need to fit the ReleaseResult structure?

Would I need to add the metadata at the Release level, or can I just populate the repo and homepage data at the ReleaseResult level?

viceice commented

normally it would be enough to have it on package level. but if it can change on different versions, it needs to be on the release.

i would prefer to have it on package level to reduce data size.

  • 👍1
jessehouwing commented

I think I can normalize most tasks to have the metadata at the package level.

Some tasks however, have been forked or transferred to a different publisher, in which case the metadata would have changed at some point. The flutter task is an example of that. It started out from:

Then got forked and taken over by:

This means that while the taskname sometimes stays the same between these extensions, the taskid and longtaskname will be different between them. So the taskid and longtaskname would be straightforward and should be able to use the latest version metadata, the shortname can be problematic.

@viceice Can the SourceUrl or the Homepage be set at the package level and the only where it need to differ at the release level? Or, since this would be for older versions in almost all cases, simply follow the latest version?

eg:

    "flutter": {
      "sourceUrl": "https://github.com/Hey24Sheep/Flutter",
      "homepage": "https://marketplace.visualstudio.com/items?itemName=Hey24Sheep.Flutter"
      releases: [
         {
             "version":  "0.3.10"
         },
         ...
         ...
         {
             "version:"  "0.2.39"
             "sourceUrl": "https://github.com/aloisdeniel/Flutter"
         },
    }
jessehouwing commented

Here's a generated sample json file containing:

  • releaseTimeStamp
  • soucreUrl
  • homepage
  • isStable
  • isDeprecated
  • downloadUrl

This amounts to +- 50MB of json.

Without the downloadUrl it's about 20MB.

https://github.com/renovatebot/azure-devops-marketplace/tree/features/preview-and-deprecation

viceice commented

i think we don't need the download url

viceice commented

what about creating a JSON file per taskid and a meta file with mapping only. then the renovate datasources first fetches the metadata file and the taskid files on demand via GitHub CDN.

that way we don't need to update renovate to get newer data.

jessehouwing commented

Sure a possibility. We now have the data and can slice and dice it in any way you want :).

jessehouwing commented

An even better option would be to switch over top an API based approach, the TFS/Azure DevOps API has all the information needed for installed tasks:

On top of the data the server already knows, the marketplace data can help people find newer versions that weren't installed into their server.

For the built-in tasks it usually doesn't make much sense to report availability of versions the server doesn't have, since they only get updated through server upgrades or Microsoft's provisioning process and there isn't much an admin or user can do unless they go the "I'll install them myself route". I have a another project to give admins of Azure DevOps Server the ability to overwrite tasks:

JamieMagee commented

@viceice I like that idea. We use GitHub pages as a JSON API.

@jessehouwing we can still implement the API-based datasource, and prefer it whenever Renovate is running in an Azure DevOps context. For the case where code is hosted in GitHub, but builds are run on Azure DevOps (this is the case for almost all the Microsoft org on GitHub for compliance reasons) we can use the cached data from this repo.

jessehouwing commented

I was thinking we could add additional config to make the API work even in a GitHub context. The Docs for configuring renovateme on Azure DevOps also show how to add GitHub configuration to generate release notes, this would be similar but the other way round.

Eg

--host-rules="[{\"domainName\":\"dev.rzure.com/myorg\",\"token\":\"$(AZURE_DEVOPS_TOKEN)\"}]"
JamieMagee commented

Yeah, good point. That's another good way to provide an API token.

jessehouwing commented

So the idea is to generate a json file per identifier and stick the ReleaseRef object in that file, right? That should be relatively simple to do from where we are now.

viceice commented

yes please. than update the renovate datasource to fetch from github cdn instead of buildin big fat json.

viceice commented

I've created a feature issue on renovate side for required datasource changes.

jessehouwing commented

There were a few illegal characters in the identifiers, I solved that with:

$identifier.ToLowerInvariant() -replace "[^a-z0-9_.-]", "_"

for now.

  • 👍1
jessehouwing commented

@JamieMagee any news on the front-end implementation of this?