Issue when configuring Artifactory mirror: cannot get list from mirror

[fcatacut]

My company has created a mirror of https://releases.hashicorp.com/terraform in Artifactory and intends to block direct downloads from the HashiCorp site. I honestly don't know if my issue is caused by how we've configured Artifactory or if it's with tfswitch itself. I lean towards the former since I don't see any issues reporting this error.

For some reason, running tfswitch --mirror https://repo.<redacted>.com/artifactory/hashicorp-releases --list-all results in a "cannot get list from mirror" error.

$ tfswitch --mirror https://repo.<redacted>.com/artifactory/hashicorp-releases --list-all
Cannot get list from mirror: https://repo.<redacted>.com/artifactory/hashicorp-releases
Use the arrow keys to navigate: ↓ ↑ → ←
? Select Terraform version:
▸ 1.3.4 *recent
  1.5.1 *recent
  1.5.0 *recent

However, I'm not running into this issue when using tfenv.

$ TFENV_REVERSE_REMOTE=1 tfenv list-remote
1.5.1
1.5.0
1.4.6
1.3.4
1.2.1

I'd like to continue using tfswitch. How should I go about troubleshooting this issue?

[tusv]

I think the issue is with how regex is being used to find versions.
On official terraform mirror a sample line in body looks like
<a href="[/terraform/1.5.1/](https://releases.hashicorp.com/terraform/1.5.1/)">terraform_1.5.1</a>
Whereas, in artifactory it may be like this
<a href="[1.5.1/](https://repo.redacted.com/artifactory/terraform/1.5.1/)">1.5.1/</a>
regex being used is \/(\d+\.\d+\.\d+)\/?" which works for the former because of leading slash and does not for the latter since there is no leading slash. https://github.com/warrensbox/terraform-switcher/blob/master/lib/list_versions.go#L34

Wonder if we can solve it by making leading slash optional in regex. i.e. \/?(\d+\.\d+\.\d+)\/?"

Not sure if there will be other issues while forming the URL for actual download.

[timdzbank]

Hi,

we can confirm the problem with the missing "/".

BG
Tim

[tusv]

Fixing regex is pretty straightforward but there is another problem. tfswitch tries to find a version before downloading it. This does not work for artifactory since it does not have list of all versions to begin with. Instead it downloads as and when requests are made. We could possibly skip checking if version exists based on some criteria/parameter.

[fcatacut]

Fixing regex is pretty straightforward but there is another problem. tfswitch tries to find a version before downloading it. This does not work for artifactory since it does not have list of all versions to begin with. Instead it downloads as and when requests are made. We could possibly skip checking if version exists based on some criteria/parameter.

Pardon my ignorance of how this app works. Isn't this what the GetTFList method is for? If someone runs tfswitch 1.5.1 --mirror https://repo.<redacted>.com/artifactory/hashicorp-releases but the version isn't in Artifactory, the command should fail. It would be the responsibility of whomever is maintaining the versions in Artifactory to ensure that this version's files exist.

[timdzbank]

Fixing regex is pretty straightforward but there is another problem. tfswitch tries to find a version before downloading it. This does not work for artifactory since it does not have list of all versions to begin with. Instead it downloads as and when requests are made. We could possibly skip checking if version exists based on some criteria/parameter.

After some troubleshooting time we can confirm that behave.
Did you find a workaround for that?

[tusv]

Fixing regex is pretty straightforward but there is another problem. tfswitch tries to find a version before downloading it. This does not work for artifactory since it does not have list of all versions to begin with. Instead it downloads as and when requests are made. We could possibly skip checking if version exists based on some criteria/parameter.

Pardon my ignorance of how this app works. Isn't this what the GetTFList method is for? If someone runs tfswitch 1.5.1 --mirror https://repo.<redacted>.com/artifactory/hashicorp-releases but the version isn't in Artifactory, the command should fail. It would be the responsibility of whomever is maintaining the versions in Artifactory to ensure that this version's files exist.

I don't think so since Artifactory will correctly fetch that version if you request it using the full URL. The main difference here is that Artifactory does not have an index page with all versions available from terraform whereas registry.terraform.io has that index page. Expecting someone else to pre-fetch a new version of artifact into Artifactory seems unnecessary and slow.
IMO, there should be a way to skip version check in some cases like when the mirror does not/can not maintain the index page without additional overhead.

[tusv]

Fixing regex is pretty straightforward but there is another problem. tfswitch tries to find a version before downloading it. This does not work for artifactory since it does not have list of all versions to begin with. Instead it downloads as and when requests are made. We could possibly skip checking if version exists based on some criteria/parameter.

Pardon my ignorance of how this app works. Isn't this what the GetTFList method is for? If someone runs tfswitch 1.5.1 --mirror https://repo.<redacted>.com/artifactory/hashicorp-releases but the version isn't in Artifactory, the command should fail. It would be the responsibility of whomever is maintaining the versions in Artifactory to ensure that this version's files exist.

I don't think so since Artifactory will correctly fetch that version if you request it using the full URL. The main difference here is that Artifactory does not have an index page with all versions available from terraform whereas registry.terraform.io has that index page. Expecting someone else to pre-fetch a new version of artifact into Artifactory seems unnecessary and slow. IMO, there should be a way to skip version check in some cases like when the mirror does not/can not maintain the index page without additional overhead.

The other way to look at this may be to still perform version check but against the registry.terraform.io and when it comes to downloading point to the mirror.

[fcatacut]

The other way to look at this may be to still perform version check but against the registry.terraform.io and when it comes to downloading point to the mirror.

@tusv wouldn't that be nice if Artifactory could be configured to download the Terraform version directly from HashiCorp's site?

I'm not privy to how my company has configured Artifactory to mirror HashCorp's releases. All I know is that they have configured our mirror in such a way that we need to set TFENV_REMOTE=https://repo.redacted.com/artifactory/hashicorp-releases. They also configured https://repo.redacted.com/artifactory/hashicorp-releases/terraform which maps to https://releases.hashicorp.com/terraform. If we want to add/cache a version to Artifactory, we have to jump through a couple of hoops:

1. Open a browser, and go to https://releases.hashicorp.com/terraform.
2. Click on the link for the version.
3. Right-click on any of the listed ZIP files, and copy the link address (e.g., https://releases.hashicorp.com/terraform/1.5.0/terraform_1.5.0_darwin_amd64.zip).
4. Past the URL into the browser's address bar, replace https://releases.hashicorp.com/terraform with https://repo.redacted.com/artifactory/hashicorp-releases/terraform (e.g., https://repo.redacted.com/artifactory/hashicorp-releases/terraform/1.5.1/terraform_1.5.1_darwin_amd64.zip), and hit <return>. A dialog window to download the file should appear. Click the Cancel button. This will cause the version to be cached in Artifactory.

It's not ideal. Given the frequency of their releases, I can live with it.

[tusv]

@fcatacut I think that does look like an acceptable workaround. The nice thing is that you only have to download(or attempt to download) a single os platform zip manually for artifactory to create a new folder for that version. After that tfswitch will correctly download it for any available os platform via artifactory for that version.

[albertorm95]

I have the same error, and we have the same process as @fcatacut

[nitrocode]

Maybe an argument can be used to input an index of all the versions, either inline or from a local/remote file. The file can be maintained in the package repository (such as artifactory) or an object in s3.

This same issue affects #315 since opentofu is in github and its releases are not on a single non-paginated url like terraform.

[adam-arnold]

I had the same issue and raised #320