Add support for Terraform 0.12's Rich Value Types
metmajer opened this issue · 33 comments
Prerequisites
Put an x into the box that applies:
- This issue describes a bug.
- This issue describes a feature request.
For more information, see the Contributing Guidelines.
Description
With the upcoming v0.12, Terraform will introduce the concept of Rich Value Types. These rich value types will introduce breaking changes to terraform-docs.
Yes, let's do this, I can make some time in this madness of schedule I'm in!
FYI Terraform has now been updated to 0.12. https://www.terraform.io/upgrade-guides/0-12.html
Yes, we're aware.
While there is no native support for Terraform 0.12 in terraform-docs, there is an alternative solution which involves awk script (kudos to @osterman and @cytopia for making this).
I have updated pre-commit-terraform to v1.16.0 where this is implemented.
See this pull request to terraform-aws-vpc module for example.
I am all with you: the missing support for Terraform 0.12 is temporary. However, if we just make it work somehow by throwing away existing functionality, the solution may work for some, but not for many others. In this case, we're not talking about an easy transition and HashiCorp has already warned us about using Terraform 0.12's public APIs which may change without warning. Integrating with HCL 2.0 directly would be the official way, but is far from straight forward.
Please also keep in mind that, while we do our best giving this project the ongoing support it needs, we do this in our spare time. At the moment, we are deeply involved in projects to make our living. Again, this doesn't mean that we don't hear you.
In the meantime, please see @antonbabenko's suggestion above. If he, as a maintainer of many Terraform community modules and a long-time user of terraform-docs can live with this temporary solution, you will hopefully too for the time being.
@moatra has brought up the idea of using the hashicorp/terraform-config-inspect module in #113. The nice thing about this approach is that it provides an abstraction over HCL and HCL2 parsing of Terraform modules and could become an ideal integration layer for us if we can address the following, current shortcomings:
- Variables are not parsed in the order they are defined. This is important as some users use terraform-docs to automatically populate forms that trigger the deployment of a Terraform module.
- Lead comments for a module, as well as comments for variables and outputs are not supported. While not a widely adopted feature, we'll need this to be backwards compatible.
- The fact that the library only supports the parsing of a single module is not an issue and is in line with the mission of terraform-docs (although multi-module support has historically been supported in some places, this should be avoided).
I'll investigate and prepare feature requests for this library as needed. Let me know if you want to join me @khos2ow @moatra @sjauld @antonbabenko ?
I've submitted a few PRs to the terraform-config-inspect module. While digging around, it looks like the module exposes SourcePos fields with filename and line number on resource, data, variable, output, and module blocks.
Variables are not parsed in the order they are defined.
It should be possible to use line numbers to do a manual sort. What's the current behavior for variables defined in different files, though?
Lead comments for a module, as well as comments for variables and outputs are not supported.
It probably won't be too onerous to add a second parsing pass looking for comments above their positions.
If you're okay with maintaining a custom parser for comments, then I don't think any upstream changes are necessary.
@moatra Great idea to have variables sorted based on line numbers. Existing use-cases assume that module variables are defined in a variables.tf only. However, we could sort by filename and line number to support modules with variables distributed across multiple files. Both HCL and HCL 2 already support the gathering of comments. Ideally, we'll re-use those within terraform-config-inspect.
With this approach, we could massively simplify support for existing and future Terraform versions. @moatra happy that you brought up this library, it will save us a ton of work. Unfortunately, the project states that public PRs are not accepted. I am not super confident, but let's see if they respond in the next days. If not, we'll have to work with a fork. Good to see that its maintainers are picking up your conversation!
Any news to support terraform 0.12?
All the open issues are linked in this thread.
I see approximately 20 issues linked from this thread, but it's really not clear which ones are relevant to the ongoing work. Would it be possible for somebody more familiar to summarize the status? I've been following this issue for over a month, but I do not understand the status.
Status update: we have decided to build the Terraform 0.12 support on top of hashicorp/terraform-config-inspect. The reasoning behind this is explained here: #62 (comment).
Check out the pull requests @moatra has started there. There's also an open discussion about gathering comments in this issue: hashicorp/terraform-config-inspect#21.
@metmajer Just noticed that the issue mentioned in the previous comment was closed two weeks ago. Is there a new plan?
Thanks for the update, I am really looking forward to 0.12 support. I'm very grateful for this tool
+1! Waiting for 12 support.
Same
Guys, check this for TF 0.12.x Docs: https://hub.docker.com/r/cytopia/terraform-docs
Guys, check this for TF 0.12.x Docs: https://hub.docker.com/r/cytopia/terraform-docs
Dockerized version
The GitHub repository for that mentioned Docker image image can be found here: https://github.com/cytopia/docker-terraform-docs
What it does, is to internally modify *.tf files of Terraform >= 0.12.x to a syntax that works with Terraform <= 0.11.x and simply runs terraform-docs over it.
The magic happens by this awk script: https://github.com/cytopia/docker-terraform-docs/blob/master/data/terraform-docs.awk
Two most well-known projects
There are two other repositories available I know of where I submit the latest changes of the awk script to:
- cloudposse/build-harness#174 (
terraform-docs.awkoriginally developed here) - antonbabenko/pre-commit-terraform#65 (mentioned above in this issue: #62 (comment))
the awk hack "works" but is not ideal. for example the usage comment is now being ignored (which i really liked), also complex types are not being added to the docs, it adds object
what we have is better than no docs but i still would like a full on support for this project
i have this:
/**
* ## IAM Role
*
* Module usage:
*
* ```
* module "iam_role" {
* source = "git@github.com:org/repo.git//modules/iam/role"
*
* name = "${var.name}"
* policy = "${var.policy}"
* mfa = true
* }
* ```
*/
on top of my main.tf in my IAM role module
but in my README i have only ## Inputs and ## Outputs
when i use terraform-docs and TF 0.11 i have my intro at the beginning
## IAM Role
Module usage:
```
module "iam_role" {
source = "git@github.com:intercom/terraform-modules.git//modules/iam/role"
name = "${var.name}"
policy = "${var.policy}"
mfa = true
}
```
## Inputs
| Name | Description | Type | Default | Required |
.......
## Outputs
| Name | Description |
|------|-------------|
...........
@khos2ow do you need a hand? My team are about to embark on a big 0.12 migration and I'm sure that we can give some Dev effort to this, seeing as how much easier it will make documentation. Just let us know what I can do.
That's awesome @rvodden . What I was thinking is to finalize some of the outstanding issues and cut a release (v0.7.0) and then merge #113 to the interim branch on the main repo, as HEAD has diverged from that PR and rebasing is gonna be difficult, and then eventually working on multiple v0.8.0-rc versions trying to iterate over that PR to finalize 0.12 support.
Makes sense. Feels like the most value we can add is to look at getting some of that re-base done ahead of time. I'll have a look tomorrow. Obviously feel free to suggest another direction if you think we could be more helpful in other places :-)
@rvodden @khos2ow
My understanding is that terraform-docs and hcltemplate have the same requirement in regards to getting to the description. Maybe both projects could benefit from enhancing the same library (cty - https://github.com/zclconf/go-cty)?
the
awkhack "works" but is not ideal. for example the usage comment is now being ignored (which i really liked), also complex types are not being added to the docs, it addsobjectwhat we have is better than no docs but i still would like a full on support for this project
I've written a Python script which wraps terraform-docs and converts the input much like @cytopia awk script does and then runs terraform-docs on those converted files. My conversion of files preserves all lines (so would preserve your usage comments) and also preserves the type information for object types as long as they don't contain any object types themselves.
Any options/arguments you pass to terraform-docs12 will be passed through to the underlying terraform-docs call.
@marcuscaisey, out of curiosity, would you be okay with putting something based on that in terraform-docs?
EDIT:
Nevermind, sorry, I just saw the PR for support for 0.12 anyway.
FYI to the folks on this thread I've picked up #113 and working on rebasing and fixing the issues, I should be having a workable PR by the end of week and my plan is to potentially cut a release sometime before end of year. Thanks for your patience and support, we will definitely have support for Terraform 0.12 by start of 2020!
Better than the gifts sent by 🎅