Terraform crashes when updating a spinnaker_project in-place. Destroying a project fails with Error: 500

Question

Terraform crashes when updating a spinnaker_project in-place. Destroying a project fails with Error: 500

Opened this issue 3 years ago · 2 comments

WHAT

Creating a spinnaker_project with terraform-provider-spinnaker is working, but managing that project afterwards is failing.

When trying to update a project in-place, terraform crashes:

Terraform will perform the following actions:

  # spinnaker_project.projects["corporate"] will be updated in-place
  ~ resource "spinnaker_project" "projects" {
      ~ email = "notanemail@email.com" -> "wwalters@email.com"
        id    = "corporate"
        name  = "corporate"
    }

  # spinnaker_project.projects["customercare"] will be updated in-place
  ~ resource "spinnaker_project" "projects" {
      ~ email = "notanemail@email.com" -> "wwalters@email.com"
        id    = "customercare"
        name  = "customercare"
    }

Plan: 0 to add, 2 to change, 0 to destroy.

----------------------------------------

spinnaker_project.projects["customercare"]: Modifying... [id=customercare]
spinnaker_project.projects["corporate"]: Modifying... [id=corporate]

Error: rpc error: code = Unavailable desc = transport is closing

Error: rpc error: code = Unavailable desc = transport is closing

panic: interface conversion: interface {} is nil, not int
2021-03-11T13:51:23.041Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: 
2021-03-11T13:51:23.041Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: goroutine 58 [running]:
2021-03-11T13:51:23.041Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: github.com/mercari/terraform-provider-spinnaker/spinnaker/api.NewCreateApplicationTask(0xc000654c80, 0x49fa6b, 0xf9dfa0, 0xc0006fb100)
2021-03-11T13:51:23.041Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    github.com/mercari/terraform-provider-spinnaker/spinnaker/api/application.go:53 +0x11a5
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: github.com/mercari/terraform-provider-spinnaker/spinnaker.resourceSpinnakerProjectUpdate(0xf9e020, 0xc000659020, 0xc000654c80, 0xd9ae20, 0xc00018e070, 0xc000397bd0, 0xacc86a, 0xc000331100)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    github.com/mercari/terraform-provider-spinnaker/spinnaker/resource_project.go:128 +0x45
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xc000181380, 0xf9dfa0, 0xc0006fb100, 0xc000654c80, 0xd9ae20, 0xc00018e070, 0x0, 0x0, 0x0)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.0/helper/schema/resource.go:309 +0x1ea
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000181380, 0xf9dfa0, 0xc0006fb100, 0xc0000b1570, 0xc000331100, 0xd9ae20, 0xc00018e070, 0x0, 0x0, 0x0, ...)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.0/helper/schema/resource.go:404 +0x390
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc00029eda0, 0xf9dfa0, 0xc0006fb100, 0xc000666410, 0xc0006fb100, 0x0, 0xfa8f40)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.0/helper/schema/grpc_provider.go:955 +0x8cf
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ApplyResourceChange(0xc000653e00, 0xf9dfa0, 0xc0006fb100, 0xc0000b13b0, 0xc000653e00, 0xc00065eea0, 0xc0004dbba0)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    github.com/hashicorp/terraform-plugin-go@v0.2.0/tfprotov5/server/server.go:331 +0xae
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0xe37fe0, 0xc000653e00, 0xf9e060, 0xc00065eea0, 0xc000658480, 0x0, 0xf9e060, 0xc00065eea0, 0xc0006b0d20, 0xef)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    github.com/hashicorp/terraform-plugin-go@v0.2.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:380 +0x214
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001fea80, 0xfa6dc0, 0xc0004ee600, 0xc00046cf00, 0xc000080c00, 0x146c8a0, 0x0, 0x0, 0x0)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    google.golang.org/grpc@v1.34.0/server.go:1210 +0x522
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: google.golang.org/grpc.(*Server).handleStream(0xc0001fea80, 0xfa6dc0, 0xc0004ee600, 0xc00046cf00, 0x0)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    google.golang.org/grpc@v1.34.0/server.go:1533 +0xd05
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000036050, 0xc0001fea80, 0xfa6dc0, 0xc0004ee600, 0xc00046cf00)
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    google.golang.org/grpc@v1.34.0/server.go:871 +0xa5
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0: created by google.golang.org/grpc.(*Server).serveStreams.func1
2021-03-11T13:51:23.042Z [DEBUG] plugin.terraform-provider-spinnaker_v0.3.0:    google.golang.org/grpc@v1.34.0/server.go:869 +0x1fd
2021/03/11 13:51:23 [DEBUG] spinnaker_project.projects["corporate"]: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2021/03/11 13:51:23 [TRACE] EvalWriteState: recording 0 dependencies for spinnaker_project.projects["corporate"]
2021/03/11 13:51:23 [TRACE] EvalWriteState: writing current state object for spinnaker_project.projects["corporate"]
2021-03-11T13:51:23.045Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021/03/11 13:51:23 [TRACE] EvalApplyProvisioners: spinnaker_project.projects["corporate"] is not freshly-created, so no provisioning is required
2021/03/11 13:51:23 [TRACE] EvalWriteState: recording 0 dependencies for spinnaker_project.projects["corporate"]
2021/03/11 13:51:23 [TRACE] EvalWriteState: writing current state object for spinnaker_project.projects["corporate"]
2021/03/11 13:51:23 [TRACE] vertex "spinnaker_project.projects[\"corporate\"]": visit complete
2021/03/11 13:51:23 [DEBUG] spinnaker_project.projects["customercare"]: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2021/03/11 13:51:23 [TRACE] EvalWriteState: recording 0 dependencies for spinnaker_project.projects["customercare"]
2021/03/11 13:51:23 [TRACE] EvalWriteState: writing current state object for spinnaker_project.projects["customercare"]
2021/03/11 13:51:23 [TRACE] EvalApplyProvisioners: spinnaker_project.projects["customercare"] is not freshly-created, so no provisioning is required
2021/03/11 13:51:23 [TRACE] EvalWriteState: recording 0 dependencies for spinnaker_project.projects["customercare"]
2021/03/11 13:51:23 [TRACE] EvalWriteState: writing current state object for spinnaker_project.projects["customercare"]
2021/03/11 13:51:23 [TRACE] vertex "spinnaker_project.projects[\"customercare\"]": visit complete
2021/03/11 13:51:23 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2021/03/11 13:51:23 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/mercari/spinnaker\"] (close)" errored, so skipping
2021/03/11 13:51:23 [TRACE] dag/walk: upstream of "root" errored, so skipping
2021-03-11T13:51:23.048Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/mercari/spinnaker/0.3.0/linux_amd64/terraform-provider-spinnaker_v0.3.0 pid=953 error="exit status 2"
2021/03/11 13:51:23 [TRACE] statemgr.Filesystem: creating backup snapshot at terraform.tfstate.backup
2021/03/11 13:51:23 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 52
2021/03/11 13:51:23 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2021/03/11 13:51:23 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2021/03/11 13:51:23 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
2021-03-11T13:51:23.078Z [DEBUG] plugin: plugin exited



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain 
sensitive information that must be redacted before it is safe to share 
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Also, destroying a project fails with Error: 500:

Terraform will perform the following actions:

  # spinnaker_project.projects["corporate"] will be destroyed
  - resource "spinnaker_project" "projects" {
      - email = "notanemail@email.com" -> null
      - id    = "corporate" -> null
      - name  = "corporate" -> null
    }

  # spinnaker_project.projects["customercare"] will be destroyed
  - resource "spinnaker_project" "projects" {
      - email = "notanemail@email.com" -> null
      - id    = "customercare" -> null
      - name  = "customercare" -> null
    }

Plan: 0 to add, 0 to change, 2 to destroy.

-----------------------------------------

spinnaker_project.projects["customercare"]: Destroying... [id=customercare]
spinnaker_project.projects["corporate"]: Destroying... [id=corporate]

Error: 500 

Error: 500

WHY

We discovered this issue after trying to update a project with new applications after it was created. Without projects, we find that it's difficult to discern which applications belong to a particular development squad in the UI.

This is an example of the relevant code being used when we encountered this issue:

# customercare.tf
locals {
  project = {
    customercare : {
      email        = local.email
      applications = keys(local.applications)
    }
  }
}

# corporate.tf
locals {
  project = {
    corporate : {
      email        = local.email
      applications = keys(local.applications)
    }
  }
}

# main.tf
locals {
  corporate_project = module.corporate.project
  customercare_project = module.customercare.project

  all_projects = merge(
    local.corporate_project,
    local.customercare_project,
  )
}

resource "spinnaker_project" "projects" {
  for_each = tomap(local.all_projects)

  name = each.key
  email = each.value.email
  config {
    applications = each.value.applications
  }
}

Answer 1 · 2021-04-09T02:07:08.000Z

Sorry for the late reply, I will take a look today 🙇

Answer 2 · 2021-04-09T15:26:16.000Z

@KeisukeYamashita That's great!

Please let me know if you need any more information. We're still creating projects with the provider, but using a lifecycle block to ignore changes to them afterwards. Would be great to keep them updated with the provider as we add and remove applications 🙂