talosctl genconfig complains that amdgpu-firmware is not a supported talos extension

Defined amdgpu-firmware in my schematic extension block:

schematic: &schematic
  customization:
    systemExtensions:
      officialExtensions:
        - siderolabs/amdgpu-firmware

When talosctl genconfig is executed, the following error is displayed:

There are issues with your talhelper config file:
field: "nodes[0].schematic"
  * "siderolabs/amdgpu-firmware" is not a supported Talos extension

It is listed under Firmware in the Extension Catalog
https://github.com/siderolabs/extensions

@dmfrey thanks for reporting! I don't know why but the supported extensions list I'm using is from https://github.com/siderolabs/image-factory#get-versionversionextensionsofficial and amdgpu-firmware is also not in the list. I will ask Talos developers about this.

Seems like with the release of Talos v1.6 there are a lot more official extensions and amdgpu is there too. I think I should use the API to get list of supported extensions from talos version but I'm currently not having free time to code. So this might take some time, if anybody would like to create a PR, it will be great.

@budimanjojo if you point me in a direction i can take a look

@dmfrey Seems like the API is internal only (not exported by the image-factory go library 😢) https://github.com/siderolabs/image-factory/blob/25fc50d09f0be71b5c65cac92191b94136f568f1/internal/artifacts/manager.go#L169

So, the only way to get the list of official extensions for the version is to do GET request to https://factory.talos.dev in /version/<version string>/extensions/official. I don't feel like making talhelper doing that on every run so I think we should fetch the list in a GitHub workflow or a script and put them in a json file that will be read on build time.

Right now, the supported extension list is a constant variable

talhelper/pkg/config/defaults.go

Line 13 in 24ce98e

var OfficialExtensions = []string{

. I think this is gonna be very ugly fix unfortunately unless they let us use the API.

The easy fix is to just put everything inside the OfficialExtensions constant and don't care about if the extension is supported by the Talos version. i.e the amdgpu-firmware shouldn't be supported by Talos <v1.6.0 but we just assume it does. It's very ugly but that's the easiest way to go in the meantime.

@dmfrey This should be fixed with the PR linked above.

You can get all supported extensions for all versions with this quick script:

for version in $(crane ls ghcr.io/siderolabs/extensions | grep -E '^v'); do
    echo "Extensions for versions ${version}"
    crane export ghcr.io/siderolabs/extensions:${version} | tar x -O image-digests
    echo
done

@goproslowyo thank you! That's so cool, it even shows everything from alpha to beta versions until v1.2.0. Seems like I can make use of that instead of using my ugly helper program.

Wrote some quick code to parse the list of extensions, not sure if it's much help @budimanjojo

talos-extensions.json
version-extension-test.go.txt

$ go build version-extension-test.go
$ ./version-extension-test v1.6.0 amd
Found strings:
- ghcr.io/siderolabs/amdgpu-firmware:20231111@sha256:4f0b95b466ba29e78a688114369e38a358a6f5ca7399b29cca67aed9aa8d2b62
- ghcr.io/siderolabs/amd-ucode:20231111@sha256:ebf5b6e47b58044398c1d83bf7c78f671fb841907411ce4bc19f45043c60f83f

package main

import (
	"encoding/json"
	"fmt"
	"os"
	"strings"
)

func isExtensionInVersionMap(version string, inputString string) ([]string, error) {
	jsonData, err := os.ReadFile("talos-extensions.json")
	if err != nil {
		return nil, err
	}
	var data map[string][]string
	if err := json.Unmarshal(jsonData, &data); err != nil {
		return nil, err
	}

	stringsInVersion, versionExists := data[version]
	if !versionExists {
		return nil, fmt.Errorf("version %s not found in the map\n", version)
	}

	var foundStrings []string
	for _, str := range stringsInVersion {
		if strings.Contains(str, inputString) {
			foundStrings = append(foundStrings, str)
		}
	}

	return foundStrings, nil
}

func main() {
	if len(os.Args) != 3 {
		fmt.Println("Usage: ./version-extension-test version extension-name\n$ ./version-extension-test v1.6.0 amdgpu")
		os.Exit(1)
	}

	version := os.Args[1]
	talosExtension := os.Args[2]

	foundStrings, err := isExtensionInVersionMap(version, talosExtension)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(1)
	}

	if len(foundStrings) > 0 {
		fmt.Println("Found strings:")
		for _, str := range foundStrings {
			fmt.Println("-", str)
		}
	} else {
		fmt.Println("No matching strings found.")
	}
}

@goproslowyo Thank you so much! I'm very interested in the talos-extensions.json above, how is it generated? If only you can improve this: https://github.com/budimanjojo/talhelper/blob/master/hack/version-schema-gen/main.go to generate the result like the json file above (minus the ghcr.io/ prefix and version suffix), it would be very helpful! Are u willing to create a PR?

The ugliest bash you've ever seen can do it.... How's this?

#!/bin/bash

output_file="talos-extensions.json"
echo "{" >"$output_file"

versions=($(crane ls ghcr.io/siderolabs/extensions | grep -E '^v'))
selected_versions=($(shuf -e "${versions[@]}" -n 2))

first=true
for version in "${selected_versions[@]}"; do
    echo -e "$([[ $first = true ]] && echo -n '' || echo -n ',')" >>"$output_file"
    echo -e "  \"$version\": [" >>"$output_file"

    firstImage=true
    crane export ghcr.io/siderolabs/extensions:$version | tar x -O image-digests | while read -r image; do
        echo -e "$([[ $firstImage = true ]] && echo -n '' || echo -n ',')" >>"$output_file"
        echo -e "    \"$image\"" >>"$output_file"
        firstImage=false
    done

    echo -e "  ]" >>"$output_file"
    first=false
done

echo "}" >>"$output_file"
jq . "$output_file" > "$output_file.tmp" && mv -f "$output_file.tmp" "$output_file"
echo "Output saved to $output_file"

If only you can improve this: master/hack/version-schema-gen/main.go to generate the result like the json file above (minus the ghcr.io/ prefix and version suffix), it would be very helpful! Are u willing to create a PR?

Yes give me like, 30m to an hour or so.

@goproslowyo ah okay so it's just a bash script with required crane to be available right? I think this can be done just inside a github workflow?

@budimanjojo Great. I can test it out whenever you like.

You can try the latest version :)

I'll try it once homebrew gets an updated image

@budimanjojo I just update to talhelper 1.16.3 and I'm still seeing

There are issues with your talhelper config file:
field: "nodes[0].schematic"
  * "siderolabs/amdgpu-firmware" is not a supported Talos extension

@dmfrey what is the talosVersion in your talconfig.yaml?

As you can see in this schema file https://github.com/budimanjojo/talhelper/blob/master/pkg/config/schemas/extensions-version-schema.json, not all versions of Talos supports siderolabs/amdgpu-firmware extension (currently only v1.6.0 above supports it)

@dmfrey what is the talosVersion in your talconfig.yaml?

Ah, gotcha. I'm on 1.5.5.

maybe I should also make it clear in the error like: "siderolabs/amdgpu-firmware" is not a supported Talos extension for v1.5.5 so it's not really confusing to the user.

@budimanjojo hopefully you should be able to pick apart all the methods used in main and use as needed.

#266