Plugin crash when cluster takes longer than hour
andrew-s opened this issue · 8 comments
andrew-s commented
I'm not sure why, but cluster creation took over an hour and so the plugin timed out (the cluster still didn't build). However, when re-running the terraform to get the state from a 'building' state;
Prior run;
module.civo_kubernetes.civo_kubernetes_cluster.primary: Still creating... [59m41s elapsed]
module.civo_kubernetes.civo_kubernetes_cluster.primary: Still creating... [59m51s elapsed]
module.civo_kubernetes.civo_kubernetes_cluster.primary: Still creating... [1h0m1s elapsed]
Error: error waiting for cluster (**cluster-id**) to be created: timeout while waiting for state to become 'ACTIVE' (last state: 'BUILDING', timeout: 1h0m0s)
This then resulted in;
The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.
Stack trace from the terraform-provider-civo_v1.0.18 plugin:
panic: runtime error: index out of range [0] with length 0
goroutine 93 [running]:
github.com/civo/terraform-provider-civo/civo.flattenNodePool(0xc00000b680, 0xf3adab, 0xb, 0xddde00)
/home/runner/work/terraform-provider-civo/terraform-provider-civo/civo/resource_kubernetes_cluster.go:484 +0x579
github.com/civo/terraform-provider-civo/civo.resourceKubernetesClusterRead(0x10d4e78, 0xc000189440, 0xc0005ee780, 0xf31620, 0xc000382640, 0xc0006d4340, 0xc00061b908, 0x40e0f8)
/home/runner/work/terraform-provider-civo/terraform-provider-civo/civo/resource_kubernetes_cluster.go:359 +0x973
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0000feee0, 0x10d4e08, 0xc0003c4480, 0xc0005ee780, 0xf31620, 0xc000382640, 0x0, 0x0, 0x0)
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource.go:347 +0x17f
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0000feee0, 0x10d4e08, 0xc0003c4480, 0xc000384070, 0xf31620, 0xc000382640, 0xc0005dcd00, 0x0, 0x0, 0x0)
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00012e228, 0x10d4e08, 0xc0003c4480, 0xc0003c44c0, 0xc0003c4480, 0x40b965, 0xeb65e0)
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc0001391c0, 0x10d4eb0, 0xc0003c4480, 0xc000188d80, 0xc0001391c0, 0xc00040b7d0, 0xc000675ba0)
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0xeef480, 0xc0001391c0, 0x10d4eb0, 0xc00040b7d0, 0xc000667f80, 0x0, 0x10d4eb0, 0xc00040b7d0, 0xc0006c8000, 0x264)
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00043ddc0, 0x10ddd58, 0xc00018bc80, 0xc0005e4600, 0xc0002be120, 0x15e1990, 0x0, 0x0, 0x0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00043ddc0, 0x10ddd58, 0xc00018bc80, 0xc0005e4600, 0x0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00003c250, 0xc00043ddc0, 0x10ddd58, 0xc00018bc80, 0xc0005e4600)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd
Error: The terraform-provider-civo_v1.0.18 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
I'm not sure how common this is but it seems that it isn't handled in the TF.
alessandroargentieri commented
@andrew-s did you have the chance to try the command again? Was this error systematic or happened (for some reasons) just once? Thanks for reporting!
andrew-s commented
@alessandroargentieri This was because the cluster name was too long, and some internal components failed - I retried with a shorter name at the time and it then worked, so I'm unsure if;
- The TF plugin handles responses from the API better
- Longer names are now supported
Hope that helps!
jarededwards commented
hi @alessandroargentieri , I actually just got what seems to be the same error but with a newer version of the provider (v1.0.26) i've attached the log from this execution.
apply time was much shorter for me (and my cluster name is 11 characters)
civo_kubernetes_cluster.kubefirst: Still creating... [6m20s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [6m30s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [6m40s elapsed]
╷
│ Error: Plugin did not respond
│
│ with civo_kubernetes_cluster.kubefirst,
│ on kubernetes.tf line 33, in resource "civo_kubernetes_cluster" "kubefirst":
│ 33: resource "civo_kubernetes_cluster" "kubefirst" {
│
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵
Stack trace from the terraform-provider-civo_v1.0.26 plugin:
panic: runtime error: index out of range [0] with length 0
goroutine 100 [running]:
github.com/civo/terraform-provider-civo/civo.flattenNodePool(...)
/home/runner/work/terraform-provider-civo/terraform-provider-civo/civo/resource_kubernetes_cluster.go:484
github.com/civo/terraform-provider-civo/civo.resourceKubernetesClusterRead({0xc0001c6460?, 0x19b2cf8?}, 0xc0004c0d00, {0x1877440?, 0xc0002c7360})
/home/runner/work/terraform-provider-civo/terraform-provider-civo/civo/resource_kubernetes_cluster.go:359 +0x1033
github.com/civo/terraform-provider-civo/civo.resourceKubernetesClusterCreate({0x19b2d30, 0xc00012cd80}, 0xc0004c0d00, {0x1877440?, 0xc0002c7360})
/home/runner/work/terraform-provider-civo/terraform-provider-civo/civo/resource_kubernetes_cluster.go:319 +0xd39
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0xc0003fb6c0, {0x19b2d68, 0xc00070dec0}, 0xd?, {0x1877440, 0xc0002c7360})
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.19.0/helper/schema/resource.go:707 +0x12e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc0003fb6c0, {0x19b2d68, 0xc00070dec0}, 0xc000794270, 0xc0004c0a00, {0x1877440, 0xc0002c7360})
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.19.0/helper/schema/resource.go:837 +0xa7a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc0003d0600, {0x19b2d68?, 0xc00070dda0?}, 0xc000493400)
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.19.0/helper/schema/grpc_provider.go:1021 +0xe3c
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0002948c0, {0x19b2d68?, 0xc00070d560?}, 0xc000242a80)
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.12.0/tfprotov5/tf5server/server.go:813 +0x4fc
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x184bce0?, 0xc0002948c0}, {0x19b2d68, 0xc00070d560}, 0xc000242a10, 0x0)
/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.12.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002901e0, {0x19b5810, 0xc000003ba0}, 0xc000444240, 0xc00027c2a0, 0x1e5e080, 0x0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.48.0/server.go:1295 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc0002901e0, {0x19b5810, 0xc000003ba0}, 0xc000444240, 0x0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.48.0/server.go:1636 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.48.0/server.go:932 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.48.0/server.go:930 +0x28a
Error: The terraform-provider-civo_v1.0.26 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
also, for what its worth the cluster did infact provision and a re-exeution of terraform plan shows only a cluster resource being created and the terraform apply fails with this error
│ Error: [ERR] failed to create the kubernetes cluster: DatabaseKubernetesClusterDuplicate: Failed to create a Kubernetes cluster, one already exists with that name
│
│ with civo_kubernetes_cluster.kubefirst,
│ on kubernetes.tf line 33, in resource "civo_kubernetes_cluster" "kubefirst":
│ 33: resource "civo_kubernetes_cluster" "kubefirst" {
│
╵
alejandrojnm commented
Hi, @andrew-s @jarededwards it possible to see the TF for this ?? also in what region is this ??
jarededwards commented
hi @alejandrojnm , this is in the NYC1 region here is the terraform code i executed, i tried to keep it as minimal as possible
jarededwards commented
also if its helpful, this is the version of terraform i was running
terraform version
Terraform v1.2.6
on darwin_arm64
alejandrojnm commented
Hi @jarededwards and @andrew-s we found a bug in the NYC1 region but it is already done and fixed, Please can you try again and let me know to close this issue.
jarededwards commented
hey @alejandrojnm thanks so much for the quick turn around. I was able to provision a new cluster in NYC1 this morning! thanks for looking into this so quickly. I will continue testing next week when i am back at work! looking forward to using civo more!
Plan: 3 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
civo_network.kubefirst: Creating...
civo_network.kubefirst: Creation complete after 3s [id=9f535a69-c330-495b-aadb-1dd2dfde4418]
civo_firewall.kubefirst: Creating...
civo_firewall.kubefirst: Creation complete after 1s [id=0fa68bd2-dd59-4cb6-8155-2bd0e0e989d5]
civo_kubernetes_cluster.kubefirst: Creating...
civo_kubernetes_cluster.kubefirst: Still creating... [10s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [20s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [30s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [40s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [50s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [1m0s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [1m10s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [1m20s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [1m30s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [1m40s elapsed]
civo_kubernetes_cluster.kubefirst: Still creating... [1m50s elapsed]
civo_kubernetes_cluster.kubefirst: Creation complete after 1m58s [id=]