PyHCL does not accept lists whose last entry is followed by a comma

[jammycakes]

The following is valid HCL and is in fact enforced both by Hashicorp's HCL formatter and many Terraform extensions for popular editors (e.g. VS Code, vim):

ip_ranges = [
  "1.2.3.4/32",
  "5.6.7.8/32",
]

However when parsed by PyHCL it gives an "unexpected RIGHTBRACKET" error. It appears that PyHCL has been confused by the comma after the second element "5.6.7.8/32" in the above example.

The HCL formatter enforces this last comma whenever the elements are placed on separate lines. As a workaround it is possible to put the elements on a single line; however, this is awkward for long lists or those where comments are required.

For compatibility, PyHCL should accept this trailing comma.

[virtuald]

Seems reasonable. PR's accepted, otherwise I'll get to it maybe in May.

[jammycakes]

I've done some further testing on this issue. The problem only occurs if the last entry in the list is followed by a comma then a comment. So for example, this will error:

ip_ranges = [
  "1.2.3.4/32",
  "5.6.7.8/32", # a comment
]

but this will not:

ip_ranges = [
  "1.2.3.4/32",
  "5.6.7.8/32" # a comment
]

and neither will this:

ip_ranges = [
  "1.2.3.4/32", # a comment
  "5.6.7.8/32",
]

There is a lexer test fixture for this, tests/lex-fixtures/array_comment.hcl, but running hcltool against this file results in this error message:

Line 4, column 36: unexpected RIGHTBRACKET

There is another file, tests/lex-fixtures/list_comma.hcl which does not have a comment after the final comma, and which works as expected, being parsed successfully by hcltool.

I've noted in the test fixtures that the final comma in array_comment.hcl returns COMMA from the lexer, but in list_comma.hcl, the final comma is lexed as COMMAEND.

[nap]

@virtuald @jammycakes You also get the same issue with MAP (pyhcl==0.3.5).

test = {
  key1 = "value2",
  key2 = ""
}

You get the following Traceback

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    pprint.pprint(hcl.load(fp))
  File "/usr/local/lib/python2.7/site-packages/hcl/api.py", line 51, in load
    return loads(fp.read())
  File "/usr/local/lib/python2.7/site-packages/hcl/api.py", line 62, in loads
    return HclParser().parse(s)
  File "/usr/local/lib/python2.7/site-packages/hcl/parser.py", line 295, in parse
    return self.yacc.parse(s, lexer=Lexer())
  File "/usr/local/lib/python2.7/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/local/lib/python2.7/site-packages/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/local/lib/python2.7/site-packages/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/usr/local/lib/python2.7/site-packages/hcl/parser.py", line 288, in p_error
    raise ValueError(msg)
ValueError: Line 2, column 26: unexpected COMMA

[scottbelden]

@nap Are you sure that's valid HCL? I don't think a comma is allowed (as the parser states).

[nap]

I've been doing some test with the following

terraform.tfvars

secrets = {
  app1 = {
    username = "app1_tfvars_value",
    password = "app1_tfvars_value",
    schema = "app1_tfvars_value",
    token = "app1_tfvars_value",
  },
  app2 = {
    username = "app2_tfvars_value",
    password = "app2_tfvars_value",
    schema = "app2_tfvars_value",
  },
  app3 = {
    username = "app3_tfvars_value",
    password = "app3_tfvars_value",
    schema = "app3_tfvars_value",
  },
}

main.tf

variable "secrets" {
  type = "map"

  default = {
    app1 = {
      username = "app1_main_value"
      password = "app1_main_value"
    }
  }
}

output "secrets" {
  value = "${var.secrets}"
}

output "app2_username" {
  value = "${lookup(var.secrets["app2"], "username")}"
}

test.py

import hcl

with open('terraform.tfvars', 'r') as fp:
    tfvars = hcl.load(fp)
    print tfvars['secrets']['app1']['token']

terraform apply

$ terraform apply

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

app2_username = app2_tfvars_value
secrets = {
  app1 = map[schema:app1_tfvars_value token:app1_tfvars_value username:app1_tfvars_value password:app1_tfvars_value]
  app2 = map[password:app2_tfvars_value schema:app2_tfvars_value username:app2_tfvars_value]
  app3 = map[schema:app3_tfvars_value username:app3_tfvars_value password:app3_tfvars_value]
}

python test.py

$ python test.py 
Traceback (most recent call last):
  File "test.py", line 4, in <module>
    tfvars = hcl.load(fp)
  File "/usr/local/lib/python2.7/site-packages/hcl/api.py", line 51, in load
    return loads(fp.read())
  File "/usr/local/lib/python2.7/site-packages/hcl/api.py", line 62, in loads
    return HclParser().parse(s)
  File "/usr/local/lib/python2.7/site-packages/hcl/parser.py", line 295, in parse
    return self.yacc.parse(s, lexer=Lexer())
  File "/usr/local/lib/python2.7/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/local/lib/python2.7/site-packages/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/local/lib/python2.7/site-packages/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/usr/local/lib/python2.7/site-packages/hcl/parser.py", line 288, in p_error
    raise ValueError(msg)
ValueError: Line 3, column 50: unexpected COMMA

[nap]

$ terraform --version
Terraform v0.9.11

[nap]

Works great. Thanks a lot.