Terraform CLI requests cause leaked go-routines

[lornasong]

Describe the bug

Root issue: Currently CTS usage of Terraform CLI causes a leaked go-routine each time the CLI is requested.

Symptoms: When Terraform logs are enabled, users can observe error from kill: os: process already finished" in the logs (more details / samples below)

Impact:

• Only Terraform driver (does not impact TFC driver)
• Increased resource usage as more Terraform requests are made by CTS daemon

Workarounds: no known workarounds

Prior to 0.6, users can see the "error from kill" log when CTS stops gracefully:

^C2022-05-02T15:39:43.469-0400 [INFO]  cli: signal received to initiate graceful shutdown: signal=interrupt
2022/05/02 15:39:43 error from kill: os: process already finished
2022/05/02 15:39:43 error from kill: os: process already finished
2022-05-02T15:39:43.469-0400 [INFO]  ctrl: stopping dependency monitoring
2022-05-02T15:39:43.469-0400 [INFO]  ctrl: stopping controller
2022/05/02 15:39:43 error from kill: os: process already finished
2022/05/02 15:39:43 error from kill: os: process already finished
2022/05/02 15:39:43 error from kill: os: process already finished
2022/05/02 15:39:43 error from kill: os: process already finished
2022-05-02T15:39:43.469-0400 [INFO]  api: shutdown api server
2022-05-02T15:39:43.469-0400 [INFO]  cli: graceful shutdown

With >=0.6 beta, users can additionally see the "error from kill" log after once-mode is completed due to some refactoring

2022-05-10T11:55:45.304-0400 [INFO]  ctrl: task completed: task_name=ls_testing_services
2022-05-10T11:55:45.304-0400 [INFO]  ctrl: all tasks completed once
2022/05/10 11:55:45 error from kill: os: process already finished
2022/05/10 11:55:45 error from kill: os: process already finished
2022/05/10 11:55:45 error from kill: os: process already finished
2022/05/10 11:55:45 error from kill: os: process already finished
2022/05/10 11:55:45 error from kill: os: process already finished
2022/05/10 11:55:45 error from kill: os: process already finished
2022-05-10T11:55:45.307-0400 [INFO]  registration: registering Consul-Terraform-Sync as a service with Consul: id=cts-a3807435-f398-534c-cb9e-a6c1551da9fc service_name=Consul-Terraform-Sync

Versions

Consul Terraform Sync

Confirmed seeing "error from kill" in v0.5.2, v0.4.3, 0.3.1. Likely to exist in all versions of CTS

Configuration File(s)

Click to toggle contents of config file

log_level = "info"
working_dir = "build"
consul {
  address = "localhost:8500"
}

driver "terraform" {
  log = true
}

task {
  name = "ls_testing_services"
  version = "0.0.1"
  module = "lornasong/cts_file/local"
  condition "services" {
    names = ["api", "web", "red"]
  }
}

Expected Behavior

No leaked go routines, no "error from kill" messages in logs

Steps to Reproduce

1. Start CTS with any config so long as terraform { log = true }
2. Wait for CTS to finish once-mode (in CTS binary >=v0.6-beta, observe "error from kill" logs)
3. Stop CTS
4. Observe "error from kill" logs

[lornasong]

This is fixed in terraform-exec dependency. CTS will need to update version