Not able to download modules from private hosting

Question

Not able to download modules from private hosting

samarth3112 opened this issue 2 years ago · 25 comments

My terraform structure is ->

.github/workflowfiles
infra/env/teama/storage.tf

modules/storage/main.tf
...................................

Modules are hosted at private registry .Now the issue is when resource defined in teama folder calls the modules , infracost gives out an error saying module path not found.

Error: Error loading Terraform modules: error downloading 'file:///home/runner/work/infra-cost-test/infra-cost-test/infrastructure/env/teama/.infracost/terraform_modules/teama.storage/terraform.hosting.company.com/company/storage/azurerm': source path error: (path omitted )
Error: Process completed with exit code 1.

Can someone helps by telling how to use it with private path please?

Answer 1 · 2022-07-08T07:06:45.000Z

Hey @samarth3112 thanks for raising this issue, the current version of Infracost 0.10.6 doesn't support private registry modules. See this issue infracost/infracost#1667 on the main CLI repository for more information. We've made a change to fix this, and it's currently sitting in master. It will be published with the upcoming 0.10.7 release. We're aiming to release 0.10.7 today, so this should hopefully resolve your issue. I'll ping you here when the release is published and good to use in GitHub actions.

Answer 2 · 2022-07-08T16:57:02.000Z

@samarth3112 v0.10.7 is live, you're probably already using infracost/actions/setup@v2 which automatically installs the latest patch version of 0.10, so can you please re-run the action and let us know if that works?
If your registry modules are private, you can read this new docs section that explains what is needed (notice the commented-out INFRACOST_TERRAFORM_CLOUD_TOKEN in the README in https://github.com/infracost/actions).

Answer 3 · 2022-07-19T18:08:11.000Z

Going to close this but feel free to add a comment if it's still having issues

Answer 4 · 2022-07-20T05:58:57.000Z

Sry for late reply .The call to pvt modules is failing.note we are not using ent terrform but open source one.

Error: Error loading Terraform modules: error looking up registry module teamapostgresdb.postgresql_database: Module versions endpoint returned status code 401
Error: Process completed with exit code 1.

Answer 5 · 2022-07-20T08:59:00.000Z

@alikhajeh1 : i am also trying to work my way using plan but somehow terraform show is not working with wrapper we have. Is there a way to directly covert hcl file to json , or can directly accept hcl

Answer 6 · 2022-07-20T09:30:41.000Z

hey @samarth3112,

Sry for late reply .The call to pvt modules is failing.note we are not using ent terrform but open source one.

Error: Error loading Terraform modules: error looking up registry module teamapostgresdb.postgresql_database: Module versions endpoint returned status code 401
Error: Process completed with exit code 1.

Sorry, I'm a little confused. So the module is open source but private? So is this something that your organisation hosts on GitHub?
If possible, could you post the source attribute of this module please?
Could you rerun infracost breakdown in the problem project, using the --log-level=debug flag, and post the output here

i am also trying to work my way using plan but somehow terraform show is not working with wrapper we have. Is there a way to directly covert hcl file to json , or can directly accept hcl

Again, I think I'm a little confused by what you're asking? Is the:
```
terraform plan -out tfplan.binary
terraform show -json tfplan.binary > plan.json
```
failing? And if so with what message?
There are ways to covert HCL files to JSON, but if you want a Terraform plan JSON there isn't a way outside of, using the default infracost breakdown method, which I believe is failing for you because of private modules, or using the terraform show method.

Answer 7 · 2022-07-20T10:23:00.000Z

Sorry, I'm a little confused. So the module is open source but private? So is this something that your organisation hosts on GitHub? Answer : Modules is not open source hosted on org registry .eg hosting.myorg.in . By open source i mean terrafrom cloud is not been used.

Could you rerun infracost breakdown in the problem project, using the --log-level=debug flag, and post the output here. Shall Do ..

Crux of issue is ,
The json file is not able to be parsed from hcl. Till hcl file able to generate but parsing is not happening.

orgcli -boilerplate -auth_resource 'https://org.hosting.xyz.com' -api_backend 'https://org.hosting.xyz.com/api' terraform init

orgcli -boilerplate -auth_resource 'https://org.hosting.xyz.com' -api_backend 'https://org.hosting.xyz.com/api' terraform terraform plan -out tfplan.binary

orgcli -boilerplate -auth_resource 'https://org.hosting.xyz.com' -api_backend 'https://org.hosting.xyz.com/api' terraform show -json tfplan.binary > plan.json. (Here it fails)

Answer 8 · 2022-07-20T13:13:31.000Z

orgcli -boilerplate -auth_resource 'https://org.hosting.xyz.com/' -api_backend 'https://org.hosting.xyz.com/api' terraform show -json tfplan.binary > plan.json. (Here it fails)

@samarth3112 what does it fail with?

Answer 9 · 2022-07-20T13:31:21.000Z

I was able to get the json file and pass it .Post that also now its not able to detect the file.Logs->

Run infracost breakdown --path /home/runner/work/poc-cost/plan.json --format=json --out-file=/tmp/infracost-base.json --log-level=debug
time="2022-07-20T13:17:32Z" level=debug result="{CloudEnabled:false}" func=main.loadCloudSettings file="/home/runner/work/infracost/infracost/cmd/infracost/main.go:249" enable_cloud_comment=false currency=USD sync_usage=false msg="Successfully loaded settings from Infracost Cloud"
time="2022-07-20T13:17:32Z" level=debug provider=terraform_dir func="github.com/infracost/internal/hcl.(*projectLocator).walkPaths" file="/home/runner/work/infracost/internal/hcl/parser.go:607" enable_cloud_comment=false currency=USD sync_usage=false routine=7 project_name= project_path=/home/runner/work/poc-design/plan.json msg="walking path /home/runner/work/poc-design/plan.json to discover terraform files"
Error: No valid Terraform files found at the given path, try a different directory

And just before this i had a ls to see if file exits .File plan.json is indeed available.

Answer 10 · 2022-07-20T13:50:15.000Z

@samarth3112 hmm this is weird, can you share the CI steps/yaml that you're using?

Answer 11 · 2022-07-20T13:50:50.000Z

name: terraform cost
run: |
ls
pwd
cat plan.json
infracost breakdown --path /home/runner/work/poc-design/plan.json --format=json --out-file=/tmp/infracost-base.json

Github .

i tired nearly every way and even cat that .json file , its showing fully its there but infracost breakdown is not able to see that . its weird indeed !

Answer 12 · 2022-07-20T14:04:19.000Z

@samarth3112 could you try with

run: |
    infracost breakdown --path plan.json --format=json --out-file=/tmp/infracost-base.json

Answer 13 · 2022-07-20T14:09:53.000Z

Error: No valid Terraform files found at the given path, try a different directory .Its still the same

Answer 14 · 2022-07-20T14:14:05.000Z

Since i am able to cat the outputs of json can i do something like

cat plan.json | infracost breakdown --format=json --out-file=/tmp/infracost-base.json

Answer 15 · 2022-07-20T14:20:06.000Z

@samarth3112 I think the plan.json must be invalid format, hence why infracost breakdown can't see it as a valid plan JSON file.

Does the cat of the plan json look something like this:

{
  "format_version": "0.2",
  "terraform_version": "1.0.2",
  "planned_values": {...},
  "resource_changes": [...],
  "configuration": {...}
}

Answer 16 · 2022-07-20T14:21:55.000Z

Yes its a json perfect as above . i copied the contents from cat outpust and tested locally on terminal . Its able to give cost .But issue on github worker.

Answer 17 · 2022-07-20T14:24:49.000Z

Does it req any env to be set . As on this step there is no env defined ?

Answer 18 · 2022-07-20T14:55:16.000Z

@samarth3112 what is the value for format_version in your plan.json - I need to know if this is a string or a float as well (e.g. quoted or unquoted)

Does it req any env to be set . As on this step there is no env defined ?

No it doesn't need any env defined.

A few further debugging steps to take:

could you also ll the directory and copy and paste the plan.json entry. e.g.:

-rw-r--r--  1 usr  staff   6.6K 11 Jul 13:41 plan.json

Did you create this plan.json in a separate step? It might be best to save it to the /tmp directory in the runner, as sometimes GitHub does weird things with the working dir. Save it there and see if --path /tmp/plan.json solves anything.

Answer 19 · 2022-07-20T15:18:08.000Z

Yes plan is computed in a different step .I tried ll and in /tmp

Run ls -l /tmp
total 64
prwx------ 1 runner docker 0 Jul 20 15:15 clr-debug-pipe-1740-5283-in
prwx------ 1 runner docker 0 Jul 20 15:15 clr-debug-pipe-1740-5283-out
prwx------ 1 runner docker 0 Jul 20 15:15 clr-debug-pipe-1759-5826-in
prwx------ 1 runner docker 0 Jul 20 15:15 clr-debug-pipe-1759-5826-out
check_suite_focus=true#step:9:17) plan1.json

and step

  name: caluclate cost 
  run: |
      ls -l /tmp
      chmod 777 /tmp/plan1.json
      ls -l /tmp/plan1.json
      infracost breakdown --path /tmp/plan1.json \
                          --format=json \
                          --out-file=/tmp/infracost-base.json

then also no luck

Answer 20 · 2022-07-20T15:58:11.000Z

@samarth3112 what is the value for format_version in your plan.json - I need to know if this is a string or a float as well (e.g. quoted or unquoted)

Answer 21 · 2022-07-20T16:12:01.000Z

{"format_version":"0.2","terraform_version":"1.0.11","variables":{"account_replication_type":{"value":"LRS"},"

Answer 22 · 2022-07-20T21:24:18.000Z

terraform_cost_final:
env:
working-directory: .
needs: terraform_json_convert
runs-on: ubuntu-latest

name: Terraform cost real
steps:

- 
  name: Setup Infracost
  uses: infracost/actions/setup@v2
  with:
      api-key: ${{ secrets.INFRACOST_API_KEY }}   

-
  name: download terraform json file
  uses: actions/download-artifact@v2
  with:
    name: upload_json_plan
    path: ${{ env.working-directory }}

- name: Run infracost 
  run: |
     ls -l 
     pwd 
     
     infracost breakdown --path plan1.json --format json --out-file /tmp/infracost.json
  working-directory: ${{ env.working-directory }}

This also is not working .

output

Run ls -l
total 36
-rw-r--r-- 1 runner docker 36546 Jul 20 21:36 plan1.json

/home/runner/work/poc-design/poc-design
time="2022-07-20T21:36:33Z" level=debug enable_cloud_comment=false func=main.loadCloudSettings file="/home/runner/work/infracost/infracost/cmd/infracost/main.go:249" result="{CloudEnabled:false}" currency=USD sync_usage=false msg="Successfully loaded settings from Infracost Cloud"
time="2022-07-20T21:36:33Z" level=debug provider=terraform_dir func="github.com/infracost/infracost/internal/hcl.(*projectLocator).walkPaths" file="/home/runner/work/infracost/infracost/internal/hcl/parser.go:607" currency=USD sync_usage=false project_path=/home/runner/work/poc-design/plan1.json project_name= routine=7 enable_cloud_comment=false msg="walking path /home/runner/work/poc-design/plan1.json to discover terraform files"
Error: No valid Terraform files found at the given path, try a different directory
Try setting --path to a Terraform plan JSON file. See https://infracost.io/troubleshoot for how to generate this.
Error: Process completed with exit code 1.

Answer 23 · 2022-07-21T08:45:22.000Z

hey @samarth3112, this is very bizarre. I've built you a special binary which includes some better debugging support so that we can get to the bottom of this. You'll need to manually download this binary into the "Run infracost" step, something like this should work:

curl -o infracost-build https://infracost-public-dumps.s3.amazonaws.com/builds/2107/infracost-linux-amd64
chmod +x ./infracost-build
./infracost-build breakdown --path plan1.json --format json --out-file /tmp/infracost.json --log-level debug

Then if you could post the debug logs here, that would be great, I'm looking for lines that are called from the func isTerraformPlanJSON. Thanks

Note i've only built linux binaries for this build, as I assume you're using a linux action runner, but if you need another OS let me know

Answer 24 · 2022-07-21T19:06:50.000Z

So finally able to resolve it . By our cli in the created json , two extra lines were getting added . Trimmed it using sed and then it was able to process it .
Thanks for your active support

Answer 25 · 2022-07-21T21:36:35.000Z

@tim775 did something similar to this happen with the plan.json files that the hashicorp/setup-terraform@v2 GitHub Action was generating? Is that why we added the terraform_wrapper: false when users were using that action? I wonder if we should trim the json in the CLI to avoid these issues as they're very tricky to debug...