Install AZ ML dependencies step is failing during deploy-model-training-pipeline.yml execution
Opened this issue · 4 comments
Describe the bug or the issue that you are facing
deploy-model-training-pipeline.yml was working totally fine till last Friday & suddenly we have started facing issue with it from Monday. Install AZ ML dependencies is failing with the below error:
/usr/bin/bash /home/vsts/work/_temp/azureclitaskscript1721726661839.sh
Collecting pip
Downloading pip-24.1.2-py3-none-any.whl (1.8 MB)
ERROR: Could not find a version that satisfies the requirement install (from versions: none)
ERROR: No matching distribution found for install
Please see the attached image for more information.
Steps/Code to Reproduce
- Clone & configure mlops-v2, mlops-project-template & mlops-template
- Create a new repo for a project & initialize the project repo using (https://github.com/Azure/mlops-v2/blob/main/documentation/deployguides/deployguide_ado.md#create-and-configure-a-new-ml-project-repo). Make sure to use MLOps interface as ' python-sdk-v2' & Infrastructure provider as 'Bicep'.
- Execute bicep-ado-deploy-infra.yml pipeline
- Execute deploy-model-training-pipeline.yml pipeline (this is where you can see the error)
Expected Output
There shall not be any error in the script
Versions
main branch of the repo in DevOps.
I am using Azure ML SDK V2
Which platform are you using for deploying your infrastrucutre?
Azure DevOps (ADO)
If you mentioned Others, please mention which platformm are you using?
No response
What are you using for deploying your infrastrucutre?
Bicep
Are you using Azure ML CLI v2 or Azure ML Python SDK v2
Azure ML Python SDK v2
Describe the example that you are trying to run?
Execute deploy-model-training-pipeline.yml pipeline
install-az-cli.yml in the template contains [python -m pip install -U --force-reinstall pip pip install <package_name>==]. Seems like this is causing the failure. I fixed it by modifying the commands to [ python -m pip install -U <package_name>==]. But now the next step in the execution which runs install-aml-cli.yml is failing with error:
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'
##[error]Script failed with error: WARNING: Auto upgrade failed. name 'exit_code' is not defined
Please help getting a solution.
It seems there is an issue with the pip upgrade. I will investigate this next week. Possibly an older version is causing an issue together with the cryptography module.
We are also getting same kind of error from July 17th :
Please let me know if any body any solution for this issue.
Microsoft is yet to provide any solution to this, but I found out some workaround though.
If you check the logs, "python -m pip install -U --force-reinstall pip" in mlops-templates -> templates -> aml-cli-v2 -> install-az-cli.yml is the culprit here.
Remove "python -m pip install -U --force-reinstall pip" from each line.
python -m pip install -U --force-reinstall pip
python -m pip install -U azure-cli==2.35
python -m pip install -U azure-ai-ml==1.1.0
python -m pip install -U azure-common==1.1.28
python -m pip install -U azure-core==1.26.1
python -m pip install -U azure-identity==1.10.0
python -m pip install -U azure-mgmt-core==1.3.0
python -m pip install -U azure-storage-blob==12.14.1
python -m pip install -U azure-storage-file-datalake==12.9.1
python -m pip install -U azure-storage-file-share==12.7.0
It may fail in the next step again(which is Install AML CLI V2) because of a missing package.
Add the below command to install-az-cli.yml to fix the issue:
python -m pip install --upgrade pyOpenSSL cryptography requests