Issue parsing multiple module input from file for specific JSON format

[lornasong]

Describe the bug

Users can create a task from file in 2 ways:

• Create a new task by passing a task configuration file to the Create Task CLI
• Initialize CTS with a task in the CTS configuration file

Configuration File Behaviour (Effects CLI and Config used by CTS on startup)

Users can configure more than one module input for a task. Module inputs can be configured in HCL or as JSON using an array or map (see examples below). When formatted as an array, the above ways to create a task will behave as expected. When formatted as a map, only one module input will be parsed. Depending on the task, it can cause the task to fail to be created or the task will potentially be incorrectly created with one module input.

Create Task API Behaviour

The API json schema definition only supports the map definition of the module input. Therefor, defining the module input using an array will fail on schema validation, and will not be an accepted request.

Examples

HCL reference:

  module_input "services" {
    names = ["web"]
  }
  
  module_input "consul-kv" {
    path = "my/path"
  }

Array example

    "module_input": [
        {
            "services": {
                "names": ["api"]
            }
        },
        {
            "consul-kv": {
                "path": "my/path"
            }
        }
    ],

Map example

  "module_input":
        {
            "services": {
                "names": ["api"]
            },
            "consul-kv": {
                "path": "my/path"
            }
    }

Note:

• Does not impact when only one module input is configured
• Does not impact hcl format

Versions

Consul Terraform Sync

consul-terraform-sync v0.5.0

Configuration File(s)

Click to toggle contents of config file

{
    "task": {
        "condition": {
            "schedule": {
                "cron": "*/15 * * * * * *"
            }
        },
        "module_input": 
        {
            "services": {
                "names": ["cts"]
            },
            "consul_kv": {
                "path": "my/path"
            }
        },
        "name": "task_a",
        "module": "lornasong/cts_kv_file/local",
        "version": "0.0.1"
    }
}

Expected Behavior

A task should be created as configured with both module inputs.

Actual Behavior

A task is created with only services module input.

Steps to Reproduce

via CTS start up

1. Start consul: consul agent -dev
2. Start cts with above config: consul-terraform-sync -config-file=config.json
3. Observe CTS errors and exists
4. Observe error in logs that the task is unable to be created
5. Observe in auto-generated main.tf, no consul-kv is variable is set

example CTS start up logs

2022-02-23T23:52:39.449-0500 [ERROR] cli: error initializing controller:
  error=
  | 
  | error: Missing required argument
  | 
  | on main.tf line 20, in module "task_services_4"
  | 20:module "task_services_4" {
  | 
  | The argument "consul_kv" is required, but no definition was found.
  
make: *** [run] Error 11

via CLI

1. Start consul: consul agent -dev
2. Start cts with any config: consul-terraform-sync -config-file=config.hcl
3. Create a task with the above config: consul-terraform-sync task create -config-file=config.json
4. Observe CLI errors that the task is unable to be created
5. Observe in auto-generated main.tf, no consul-kv is variable is set

example CLI error logs

on main.tf line 20, in module "task_services_4"
20:module "task_services_4" {

The argument "consul_kv" is required, but no definition was found.
, see logs for more details (Request ID: 61109185-d5fc-be77-d9af-741fce5ec662)
make: *** [create] Error 11

Workarounds

To summarize what is supported via CTS config and CLI, and what is supported via the API:

• API supports JSON only, so the module input must be a json map (see map example above)
• Config/CLI supports HCL and JSON, so if using JSON the module input must be defined as an array (see array example above). Alternatively HCL can be used and behaves as expected.

Using the appropriate definitions for the input method above will allow for the module input to be interpreted correctly, and the task will be created with the right module inputs.

[wilkermichael]

Investigation Results:

How does it affect CTS

• This affects all "block" HCL definitions in CTS, all of which will handle an array but not a map using the current decoding

How are other people using it?

• according to HCL, both both map/array definitions are acceptable, and preferably both are supported
• people seem to prefer the map definition in general
• the array definition is the most direct representation because it retains the possibility of multiple blocks with the same label and keeping the declaration order
• Consul only supports the array definition

Proposed Solution

• HCL supports the array and json definitions of a block in general, however, writing a JSON schema for our API which supports both may be cumbersome. We can still look into supporting both.
• If supporting both is not the right path, then proposal is to switch API json schema to expect an array for the module input, rather than a map. This way it would conform with the config interpretation and could easily be copied between the two. Going to the array would be a deprecation from the current API map support.