allow use of fs.FS for specifying the filesystem
PaluMacil opened this issue · 5 comments
I'm using embedded files for configuration, and this works great in Go except for the file function since it looks at the harddisk. I would love to be able to set the fs.FS used to resolve files. To keep existing behavior, the default could be DirFS, but this would enable the file function to access file embedding, S3, or a multitude of other filesystems. This would also be easier when one wants to split up configuration into multiple hcl files while embedding but having cross file references.
Hi @PaluMacil! Thanks for sharing this feature request.
As far as I can remember, all of the functions in this module which directly read from disk are just thin wrappers around functions that parse in-memory buffers, and are there only to help with the most common case of reading a single file from a fixed location on disk.
If you ignore those wrapper functions and instead use the real functions that take buffers ([]byte) as input then I expect you should be able to parse data read from any source, as long as it can produce a byte array.
Would that be sufficient for the problem you are trying to solve?
Thanks!
Are you able to call these from an hcl template? To clarify a little more, I'm talking about the file function in hcl syntax, not something in Go. I was simply thinking that instead of os.Open they could use fs.Open and we could potentially look a way to set the fs.FS. By default if it was set to DirFS, then all Opens would work like they do now.
I'm not sure where the template syntax functions live, but I think if I have a few pointers as to where to look for things, I could probably make an example MR to explain what I mean and then perhaps you could say whether you think it's valuable and probably point me to how you think it should be better implemented.
@apparentlymart Is https://developer.hashicorp.com/terraform/language/functions/file in hcl by default or am I asking about a function that is only added by Terraform specifically anyway?
Oh, I see! That function is defined in Terraform itself, so if you are implementing your own HCL-based language and want to offer a similar function then you'll need to provide your own implementation and include it in the hcl.EvalContext when you evaluate an expression.
You can find Terraform's implementation in the MakeFileFunc function in internal/lang/funcs/filesystem.go, in case you'd like to use that as a reference for your implementation. Terraform's implementation is made more complex by reusing some parts across a number of different functions, so you can probably simplify it considerably if you only need the file function and not any of Terraform's other variants.
If you want to use an arbitrary fs.FS in your implementation then I imagine you'd follow a similar pattern as Terraform does of having a function that returns a function which refers to the parameters of the outer function, but you could provide an fs.FS as the first argument, instead of passing a string containing a base directory as Terraform does.
Thank you! I had just started to read about github.com/zclconf/go-cty when I noticed your response, and you pointing me to the implementation and Terraform certainly saved me a lot of time. That's exactly what I wanted. I'll certainly be able to figure out how to add functions of my own from that.