kyma-incubator/terraform-provider-gardener

`terraform destroy` needs extra action while destroying an cluster with some custom annotations

Closed this issue · 5 comments

neo-liang-sap commented

Hello,

The scenario looks like this,

  1. . while i was trying to add an cluster without any customized annotation info in metadata section, the terraform-provider-gardener works perfectly. i can create the cluster with terraform apply and destroy with terraform destroy

for example in main.tf

  metadata {
    name      = var.name
    namespace = var.gardener_namespace
  }
  1. . when i was trying to adding an cluster with some customized annotation info in metadata, e.g.
metadata {
    name      = var.name
    namespace = var.gardener_namespace
    annotations = {
      "garden.sapcloud.io/purpose" = "development"
      "dashboard.garden.sapcloud.io/no-hibernation-schedule" = true
    }
  }

i can add the cluster without problem by terraform apply

but when i would like to destroy the cluster, if i execute terraform destroy i met errors, indicating only an cluster with annotation "confirmation.garden.sapcloud.io/deletion" = true could be deleted.

So i need to do following steps

  • a. modify the cluster main.tf like
annotations = {
    "garden.sapcloud.io/purpose" = "development"
    "dashboard.garden.sapcloud.io/no-hibernation-schedule" = true
    "confirmation.garden.sapcloud.io/deletion" = true
  }

  • b. run terraform apply to modify the cluster

  • c. run terraform destroy to delete the cluster

So i wonder are these steps work as design? or maybe somewhere in terraform-provider-gardener can be modify hence cluster with custom annotation could be deleted directly by executing terraform destroy once?

Thanks! any hint on the implementation idea would be much appreciated

clebs commented

Hi @neo-liang-sap,

that is a very interesting finding and from my limited understanding I suspect that it is a bug in gardener.

What might probably happening is that gardener is not able to edit the annotations when destroying (they are usually immutable).

I am afraid this might not be an issue on the terraform provider but on gardener itself.

I would suggest you take this up with the gardener team (https://github.com/gardener/gardener) and see what feedback you can get from them.

neo-liang-sap commented

Hi @clebs
Yes the metadata.annotations in gardener is immutable, i will bring up this issue to gardener team to see if any better solution than what i tried.
Thanks!
-Neo

vpnachev commented

I suspect that it is a bug in gardener.

@clebs - this is not a bug, it is by design so. For details see gardener/gardener#1816 (comment)

Yes the metadata.annotations in gardener is immutable

@neo-liang-sap , this is not true, too.

neo-liang-sap commented

Hi @clebs / @vpnachev

Thanks for the explanation, now i totally understand that adding "confirmation.garden.sapcloud.io/deletion" = true to shoot cluster before deletion is by design.
I have just one question regarding annotation.metadata handling logic in terraform-provider-gardener:

In my test described at this ticket, if i build an shoot without any customized metadata.annotation, the shoot contains "confirmation.garden.sapcloud.io/deletion" = true which could be deleted directly.

If i build an shoot with customized metadata.annotation, this shoot has no annotation of "confirmation.garden.sapcloud.io/deletion" = true

I checked code in terraform-provider-gardener, there are two lines explicitly adding "confirmation.garden.sapcloud.io/deletion" = true no matter whether the metadata.annotation is nil or not

  1. https://github.com/kyma-incubator/terraform-provider-gardener/blob/master/expand/expand_metadata.go#L25
  2. https://github.com/kyma-incubator/terraform-provider-gardener/blob/master/shoot/resource_shoot.go#L97

If i understand correctly, so these two lines didn't work when metadata.annotation is not nil (contains some other annotations)? correct me if i was wrong, if there's any place i can worked on to improve this i'd like to help

Thanks!
-Neo

neo-liang-sap commented

close this issue as confirmed by Gardener dev, it's working as design