Deploying Infrastructure via Azure DevOps
corticalstack opened this issue · 6 comments
Describe the bug or the issue that you are facing
Error when following the deploy with terraform guide, outer loop step 8
https://github.com/Azure/mlops-v2/blob/main/documentation/deployguides/deployguide_ado.md
When running the pipeline to create the ML infra, I get the error:
Repository mlops-templates references endpoint github-connection which does not exist or is not authorized for use
I have seen a similar issue (now closed) which indicated to install the Terraform for ADO extension. I have this installed.
Steps/Code to Reproduce
As described earlier
Expected Output
Pipeline runs successfully, provisioning the required AML resources
Versions
As described earlier
Which platform are you using for deploying your infrastrucutre?
Azure DevOps (ADO)
If you mentioned Others, please mention which platformm are you using?
No response
What are you using for deploying your infrastrucutre?
Terraform
Are you using Azure ML CLI v2 or Azure ML Python SDK v2
Azure ML Python SDK v2
Describe the example that you are trying to run?
As described earlier
I thinnk the problem is that the step-by-step guide for MLOps V2 with ADO and Terraform misses the service connection creation to create github-connection, as documented in quickstart.md for mlopsv2 repo:
3.3.5 Select "Github", select "Next", select "Personal Access Token" and paste your Github SSO Token in the Personal Access token field, in the "Service connection name" field, enter "github-connection", grant pipeline security access, then select "Save".
Repeat this step, and for the "Service connection name" enter YOUR GITHUB ORGANIZATION NAME or YOUR GITHUB NAME. Finishing this step, your conection should look like this.
@corticalstack What is the error that you are getting. The comment you made on Issue #72 was related to the terraform installer that wasn't installed.
The github-connection
is not required as your repositories are in ADO. In case you still want to store the repository in GitHub and run the pipelines in Azure DevOps (ADO), then you can setup a GitHub connection. Here are the steps that you can follow this.
- Go to your Azure DevOps project and navigate to the
Project Settings
page. - Select
Service connections
under the "Pipelines" section. - Click on the
New service connection
button and select "GitHub" (since the type of repository in your code is GitHub). - You will be prompted to enter the connection details, including the GitHub repository URL, a personal access token with the necessary permissions, and a connection name.
- After entering the connection details, click on the
Verify and Save
button to create the connection. - Once the connection is created, you can refer to it by the name you gave it in your code, in this case
github-connection
.
You can now use this endpoint in your pipelines to connect to the specified GitHub repository. That's it! Once you have set up the endpoint, you can reference it in your Azure DevOps pipeline configurations, like the code you provided earlier.
@setuc Thanks for the detailed response, which is in fact what I did to get it working and move forward. However, as you say, I'm using ADO for repo management and running the pipelines, so the question would be, "why is github-connection in the yml config for the ADO-only path, and how should an ADO-pure yml look for the resources>repositories part of the yml?"
Perhaps for the ADO-only, no endpoint key-value pair is needed under repositories in the yml?
I also encountered this issue when performing an ADO only deployment. The problem is with https://github.com/Azure/mlops-project-template/blob/main/infrastructure/terraform/devops-pipelines/tf-ado-deploy-infra.yml, but also other files too, for example:
which all contain the hardcoded endpoint
and type
keys for a github connection that should not exist in an ADO only setup:
resources:
repositories:
- repository: mlops-templates # Template Repo
name: Azure/mlops-templates # need to change org name from "Azure" to your own org
endpoint: github-connection # need to set up and hardcode
type: github
ref: main
This can be fixed in at least a couple of ways such as inline syntax similar to this reference or maybe a better approach from a resource
consistency point of view by the following:
resources:
repositories:
- repository: mlops-templates
name: mlops-templates
type: git
ref: main
I have only been testing on an ADO basis so I don't know whether the endpoint: github-connection
is required for other scenarios (I presume so). I think if it is required, then the YAML files should be either patched based on the deployment scenario, i.e. ADO or GitHub or multiple copies of the file maintained (probably less desirable).
@tonyskidmore thanks for the follow-up and info, appreciated. Frustrating to see little movement on all these open issues
This still continues to be an issue. thanks @tonyskidmore for the solution.