kyma-project/lifecycle-manager

Cache issues during the upgrade process

janmedrek opened this issue · 3 comments

janmedrek commented

Description

During the upgrade process, some resources are cached and the Manifest goes into an Error state. After the KLM pod is restarted resources are applied correctly and the module is in a Ready state.

Steps to reproduce

  1. The old version of the module is enabled (based on the old module template)
  2. The module is in a ready state
  3. The new version of the module is enabled (a new version of the module template is introduced)
  4. The module goes to an error state (due to the CRD upgrade not being possible)
  5. Removed CRs and old CRD
  6. The module goes to an error state (due to not being able to apply resources)
  7. Restarted KLM pod in the KCP
  8. The module goes to the ready/warning state

Environment Type

Managed

Environment Info

Kubernetes Version: v1.26.9
Docker Version: not relevant
Kyma CLI Version: not relevant

Attachments

Old ModuleTemplate:

apiVersion: operator.kyma-project.io/v1beta2
kind: ModuleTemplate
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: kyma-modules-fast-channel:operator.kyma-project.io/ModuleTemplate:kcp-system/api-gateway-fast
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operator.kyma-project.io/v1beta2","kind":"ModuleTemplate","metadata":{"annotations":{"argocd.argoproj.io/tracking-id":"kyma-modules-fast-channel:operator.kyma-project.io/ModuleTemplate:kcp-system/api-gateway-fast","operator.kyma-project.io/doc-url":"https://kyma-project.io/#/api-gateway/user/README","operator.kyma-project.io/is-cluster-scoped":"true","operator.kyma-project.io/module-version":"2.0.0"},"labels":{"argocd.argoproj.io/instance":"kyma-modules-fast-channel","operator.kyma-project.io/controller-name":"manifest","operator.kyma-project.io/managed-by":"lifecycle-manager","operator.kyma-project.io/module-name":"api-gateway"},"name":"api-gateway-fast","namespace":"kcp-system"},"spec":{"channel":"fast","data":{"apiVersion":"operator.kyma-project.io/v1alpha1","kind":"APIGateway","metadata":{"name":"default"},"spec":{"enableKymaGateway":true}},"descriptor":{"component":{"componentReferences":[],"labels":[{"name":"security.kyma-project.io/scan","value":"enabled","version":"v1"}],"name":"kyma-project.io/module/api-gateway","provider":"{\"name\":\"kyma-project.io\",\"labels\":[{\"name\":\"kyma-project.io/built-by\",\"value\":\"cli\",\"version\":\"v1\"}]}","repositoryContexts":[{"baseUrl":"europe-docker.pkg.dev/kyma-project/modules-internal","componentNameMapping":"urlPath","type":"OCIRegistry"}],"resources":[{"access":{"imageReference":"europe-docker.pkg.dev/kyma-project/prod/api-gateway-manager:2.0.0","type":"ociRegistry"},"labels":[{"name":"scan.security.kyma-project.io/type","value":"third-party-image","version":"v1"}],"name":"api-gateway-manager","relation":"external","type":"ociImage","version":"2.0.0"},{"access":{"imageReference":"europe-docker.pkg.dev/kyma-project/prod/external/oryd/oathkeeper:v0.38.25-beta.1","type":"ociRegistry"},"labels":[{"name":"scan.security.kyma-project.io/type","value":"third-party-image","version":"v1"}],"name":"oathkeeper","relation":"external","type":"ociImage","version":"v0.38.25-beta.1"},{"access":{"imageReference":"europe-docker.pkg.dev/kyma-project/prod/external/oryd/oathkeeper-maester:v0.1.5","type":"ociRegistry"},"labels":[{"name":"scan.security.kyma-project.io/type","value":"third-party-image","version":"v1"}],"name":"oathkeeper-maester","relation":"external","type":"ociImage","version":"v0.1.5"},{"access":{"globalAccess":{"digest":"sha256:1553f2511f6e671ba9770c6f0ebaa0d4842334005ff07deab6e26fa209f5f577","mediaType":"application/octet-stream","ref":"europe-docker.pkg.dev/kyma-project/modules-internal/component-descriptors/kyma-project.io/module/api-gateway","size":23058,"type":"ociBlob"},"localReference":"sha256:1553f2511f6e671ba9770c6f0ebaa0d4842334005ff07deab6e26fa209f5f577","mediaType":"application/octet-stream","type":"localBlob"},"name":"raw-manifest","relation":"local","type":"yaml","version":"2.0.0"}],"sources":[{"access":{"commit":"681a413f24691d77eaeb403a762d0f2d0c5d0eb0","repoUrl":"https://github.com/kyma-project/api-gateway.git","type":"gitHub"},"labels":[{"name":"git.kyma-project.io/ref","value":"HEAD","version":"v1"},{"name":"scan.security.kyma-project.io/dev-branch","value":"","version":"v1"},{"name":"scan.security.kyma-project.io/rc-tag","value":"","version":"v1"},{"name":"scan.security.kyma-project.io/language","value":"golang-mod","version":"v1"},{"name":"scan.security.kyma-project.io/exclude","value":"**/tests/**,**/test/**,**/*_test.go","version":"v1"}],"name":"module-sources","type":"Github","version":"2.0.0"}],"version":"2.0.0"},"meta":{"schemaVersion":"v2"}}}}
    operator.kyma-project.io/doc-url: https://kyma-project.io/#/api-gateway/user/README
    operator.kyma-project.io/is-cluster-scoped: "true"
    operator.kyma-project.io/module-version: 2.0.0
  creationTimestamp: "2023-12-28T12:50:23Z"
  generation: 1
  labels:
    argocd.argoproj.io/instance: kyma-modules-fast-channel
    operator.kyma-project.io/controller-name: manifest
    operator.kyma-project.io/managed-by: lifecycle-manager
    operator.kyma-project.io/module-name: api-gateway
  name: api-gateway-fast
  namespace: kyma-system
  resourceVersion: "131156"
  uid: b7ab46ce-2675-4d0a-be46-c25ef26e9ea9
spec:
  channel: fast
  data:
    apiVersion: operator.kyma-project.io/v1alpha1
    kind: APIGateway
    metadata:
      name: default
    spec:
      enableKymaGateway: true
  descriptor:
    component:
      componentReferences: []
      labels:
      - name: security.kyma-project.io/scan
        value: enabled
        version: v1
      name: kyma-project.io/module/api-gateway
      provider: '{"name":"kyma-project.io","labels":[{"name":"kyma-project.io/built-by","value":"cli","version":"v1"}]}'
      repositoryContexts:
      - baseUrl: europe-docker.pkg.dev/kyma-project/modules-internal
        componentNameMapping: urlPath
        type: OCIRegistry
      resources:
      - access:
          imageReference: europe-docker.pkg.dev/kyma-project/prod/api-gateway-manager:2.0.0
          type: ociRegistry
        labels:
        - name: scan.security.kyma-project.io/type
          value: third-party-image
          version: v1
        name: api-gateway-manager
        relation: external
        type: ociImage
        version: 2.0.0
      - access:
          imageReference: europe-docker.pkg.dev/kyma-project/prod/external/oryd/oathkeeper:v0.38.25-beta.1
          type: ociRegistry
        labels:
        - name: scan.security.kyma-project.io/type
          value: third-party-image
          version: v1
        name: oathkeeper
        relation: external
        type: ociImage
        version: v0.38.25-beta.1
      - access:
          imageReference: europe-docker.pkg.dev/kyma-project/prod/external/oryd/oathkeeper-maester:v0.1.5
          type: ociRegistry
        labels:
        - name: scan.security.kyma-project.io/type
          value: third-party-image
          version: v1
        name: oathkeeper-maester
        relation: external
        type: ociImage
        version: v0.1.5
      - access:
          globalAccess:
            digest: sha256:1553f2511f6e671ba9770c6f0ebaa0d4842334005ff07deab6e26fa209f5f577
            mediaType: application/octet-stream
            ref: europe-docker.pkg.dev/kyma-project/modules-internal/component-descriptors/kyma-project.io/module/api-gateway
            size: 23058
            type: ociBlob
          localReference: sha256:1553f2511f6e671ba9770c6f0ebaa0d4842334005ff07deab6e26fa209f5f577
          mediaType: application/octet-stream
          type: localBlob
        name: raw-manifest
        relation: local
        type: yaml
        version: 2.0.0
      sources:
      - access:
          commit: 681a413f24691d77eaeb403a762d0f2d0c5d0eb0
          repoUrl: https://github.com/kyma-project/api-gateway.git
          type: gitHub
        labels:
        - name: git.kyma-project.io/ref
          value: HEAD
          version: v1
        - name: scan.security.kyma-project.io/dev-branch
          value: ""
          version: v1
        - name: scan.security.kyma-project.io/rc-tag
          value: ""
          version: v1
        - name: scan.security.kyma-project.io/language
          value: golang-mod
          version: v1
        - name: scan.security.kyma-project.io/exclude
          value: '**/tests/**,**/test/**,**/*_test.go'
          version: v1
        name: module-sources
        type: Github
        version: 2.0.0
      version: 2.0.0
    meta:
      schemaVersion: v2
  mandatory: false

New ModuleTemplate:

apiVersion: operator.kyma-project.io/v1beta2
kind: ModuleTemplate
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operator.kyma-project.io/v1beta2","kind":"ModuleTemplate","metadata":{"annotations":{"operator.kyma-project.io/is-cluster-scoped":"false","operator.kyma-project.io/module-version":"3.0.0"},"generation":1,"labels":{"operator.kyma-project.io/module-name":"api-gateway"},"name":"api-gateway-superfast","namespace":"kcp-system"},"spec":{"channel":"superfast","data":{"apiVersion":"operator.kyma-project.io/v1alpha1","kind":"APIGateway","metadata":{"name":"default","namespace":"kyma-system"},"spec":{"enableKymaGateway":true}},"descriptor":{"component":{"componentReferences":[],"labels":[{"name":"security.kyma-project.io/scan","value":"enabled","version":"v1"}],"name":"kyma-project.io/module/api-gateway","provider":"{\"name\":\"kyma-project.io\",\"labels\":[{\"name\":\"kyma-project.io/built-by\",\"value\":\"cli\",\"version\":\"v1\"}]}","repositoryContexts":[{"baseUrl":"europe-docker.pkg.dev/kyma-project/dev/unsigned","componentNameMapping":"urlPath","type":"OCIRegistry"}],"resources":[{"access":{"imageReference":"europe-docker.pkg.dev/kyma-project/prod/api-gateway-manager:v20231214-23528ca2","type":"ociRegistry"},"labels":[{"name":"scan.security.kyma-project.io/type","value":"third-party-image","version":"v1"}],"name":"api-gateway-manager","relation":"external","type":"ociImage","version":"v20231214-23528ca2"},{"access":{"imageReference":"europe-docker.pkg.dev/kyma-project/prod/external/oryd/oathkeeper:v0.38.25-beta.1","type":"ociRegistry"},"labels":[{"name":"scan.security.kyma-project.io/type","value":"third-party-image","version":"v1"}],"name":"oathkeeper","relation":"external","type":"ociImage","version":"v0.38.25-beta.1"},{"access":{"imageReference":"europe-docker.pkg.dev/kyma-project/prod/external/oryd/oathkeeper-maester:v0.1.5","type":"ociRegistry"},"labels":[{"name":"scan.security.kyma-project.io/type","value":"third-party-image","version":"v1"}],"name":"oathkeeper-maester","relation":"external","type":"ociImage","version":"v0.1.5"},{"access":{"globalAccess":{"digest":"sha256:62dc6ac5e046fe5c0d1f1571d5aa133914c03f10b14efa03b6b0c449a4bd4d47","mediaType":"application/octet-stream","ref":"europe-docker.pkg.dev/kyma-project/dev/unsigned/component-descriptors/kyma-project.io/module/api-gateway","size":39927,"type":"ociBlob"},"localReference":"sha256:62dc6ac5e046fe5c0d1f1571d5aa133914c03f10b14efa03b6b0c449a4bd4d47","mediaType":"application/octet-stream","type":"localBlob"},"name":"raw-manifest","relation":"local","type":"yaml","version":"0.0.1-PR-774"}],"sources":[{"access":{"commit":"def7b780bd6ab43e593d841a3caeabbe8b51e83f","repoUrl":"github.com/kyma-project/api-gateway","type":"gitHub"},"labels":[{"name":"git.kyma-project.io/ref","value":"refs/heads/main","version":"v1"},{"name":"scan.security.kyma-project.io/rc-tag","value":"","version":"v1"},{"name":"scan.security.kyma-project.io/language","value":"golang-mod","version":"v1"},{"name":"scan.security.kyma-project.io/dev-branch","value":"","version":"v1"},{"name":"scan.security.kyma-project.io/subprojects","value":"false","version":"v1"},{"name":"scan.security.kyma-project.io/exclude","value":"**/tests/**,**/test/**,**/*_test.go","version":"v1"}],"name":"module-sources","type":"Github","version":"0.0.1-PR-774"}],"version":"3.0.0"},"meta":{"schemaVersion":"v2"}},"mandatory":false}}
    operator.kyma-project.io/is-cluster-scoped: "false"
    operator.kyma-project.io/module-version: 3.0.0
  creationTimestamp: "2023-12-28T14:21:46Z"
  generation: 1
  labels:
    operator.kyma-project.io/module-name: api-gateway
  name: api-gateway-superfast
  namespace: kyma-system
  resourceVersion: "131135"
  uid: 7128e0e8-6c60-4301-894a-ef353d74f344
spec:
  channel: superfast
  data:
    apiVersion: operator.kyma-project.io/v1alpha1
    kind: APIGateway
    metadata:
      name: default
      namespace: kyma-system
    spec:
      enableKymaGateway: true
  descriptor:
    component:
      componentReferences: []
      labels:
      - name: security.kyma-project.io/scan
        value: enabled
        version: v1
      name: kyma-project.io/module/api-gateway
      provider: '{"name":"kyma-project.io","labels":[{"name":"kyma-project.io/built-by","value":"cli","version":"v1"}]}'
      repositoryContexts:
      - baseUrl: europe-docker.pkg.dev/kyma-project/dev/unsigned
        componentNameMapping: urlPath
        type: OCIRegistry
      resources:
      - access:
          imageReference: europe-docker.pkg.dev/kyma-project/prod/api-gateway-manager:v20231214-23528ca2
          type: ociRegistry
        labels:
        - name: scan.security.kyma-project.io/type
          value: third-party-image
          version: v1
        name: api-gateway-manager
        relation: external
        type: ociImage
        version: v20231214-23528ca2
      - access:
          imageReference: europe-docker.pkg.dev/kyma-project/prod/external/oryd/oathkeeper:v0.38.25-beta.1
          type: ociRegistry
        labels:
        - name: scan.security.kyma-project.io/type
          value: third-party-image
          version: v1
        name: oathkeeper
        relation: external
        type: ociImage
        version: v0.38.25-beta.1
      - access:
          imageReference: europe-docker.pkg.dev/kyma-project/prod/external/oryd/oathkeeper-maester:v0.1.5
          type: ociRegistry
        labels:
        - name: scan.security.kyma-project.io/type
          value: third-party-image
          version: v1
        name: oathkeeper-maester
        relation: external
        type: ociImage
        version: v0.1.5
      - access:
          globalAccess:
            digest: sha256:62dc6ac5e046fe5c0d1f1571d5aa133914c03f10b14efa03b6b0c449a4bd4d47
            mediaType: application/octet-stream
            ref: europe-docker.pkg.dev/kyma-project/dev/unsigned/component-descriptors/kyma-project.io/module/api-gateway
            size: 39927
            type: ociBlob
          localReference: sha256:62dc6ac5e046fe5c0d1f1571d5aa133914c03f10b14efa03b6b0c449a4bd4d47
          mediaType: application/octet-stream
          type: localBlob
        name: raw-manifest
        relation: local
        type: yaml
        version: 0.0.1-PR-774
      sources:
      - access:
          commit: def7b780bd6ab43e593d841a3caeabbe8b51e83f
          repoUrl: github.com/kyma-project/api-gateway
          type: gitHub
        labels:
        - name: git.kyma-project.io/ref
          value: refs/heads/main
          version: v1
        - name: scan.security.kyma-project.io/rc-tag
          value: ""
          version: v1
        - name: scan.security.kyma-project.io/language
          value: golang-mod
          version: v1
        - name: scan.security.kyma-project.io/dev-branch
          value: ""
          version: v1
        - name: scan.security.kyma-project.io/subprojects
          value: "false"
          version: v1
        - name: scan.security.kyma-project.io/exclude
          value: '**/tests/**,**/test/**,**/*_test.go'
          version: v1
        name: module-sources
        type: Github
        version: 0.0.1-PR-774
      version: 3.0.0
    meta:
      schemaVersion: v2
  mandatory: false
janmedrek commented

We need to recheck whether recent fixes solved the problem, if that is indeed still popping up we need to create a follow-up with the details.

Tomasz-Smelcerz-SAP commented

I cannot reproduce that.
It looks like the superfast module template refers to non-existing data. This is the status of the manifest CR (from KCP-dev) after applying the superfast module template:

  lastOperation:
    lastUpdateTime: "2024-02-26T06:57:10Z"
    operation: 'failed to extract raw manifest from layer digest: failed fetching
      blob for layer europe-docker.pkg.dev/kyma-project/dev/unsigned/component-descriptors/kyma-project.io/module/api-gateway@sha256:62dc6ac5e046fe5c0d1f1571d5aa133914c03f10b14efa03b6b0c449a4bd4d47:
      GET https://europe-docker.pkg.dev/v2/kyma-project/dev/unsigned/component-descriptors/kyma-project.io/module/api-gateway/blobs/sha256:62dc6ac5e046fe5c0d1f1571d5aa133914c03f10b14efa03b6b0c449a4bd4d47:
      BLOB_UNKNOWN: Unknown blob: sha256:62dc6ac5e046fe5c0d1f1571d5aa133914c03f10b14efa03b6b0c449a4bd4d47'
  state: Error

@janmedrek Any hints on reproducing that?

janmedrek commented

Well, I see only one module template available for the API Gateway and have not heard about any further issues in this area. Let's close it and reopen if someone can reproduce it.