seeq12/seeq-azureml

developer-notebook cannot see ACI endpoint

Closed this issue · 10 comments

insm-cvx commented

Our company's seeq server needs to be upgraded for us to try out the seeq-azureml library in Seeq Data Lab, so until then, we're trying to test it locally. We think we've followed the install and setup instructions properly as the notebook can successfully authenticate to our AML workspace, but we cannot see our AML endpoint in the Model Inputs dropdown. Here's what we've tried so far:

  1. pip installed seeq-azureml in a fresh conda env
  2. Cloned down the seeq-azureml repo
  3. Trained and registered a simple model in AML with tags taken from the repo documentation
    seeq-model
  4. Deployed the model as an AciWebService with a tag of Seeq="true" (we previously tried with Seeq=True)
    seeq-endpoint
  5. Ran the Jupyter notebook and tried to select the endpoint.
    image
monstrorivas commented

Have you tried Seeq : true for the tag in your Step 4? That is, the Name of the tag is Seeq and the Value is true (pay attention to the correct capitalization). Let me know if that still doesn’t work and we can troubleshoot more

insm-cvx commented

Have you tried Seeq : true for the tag in your Step 4? That is, the Name of the tag is Seeq and the Value is true (pay attention to the correct capitalization). Let me know if that still doesn’t work and we can troubleshoot more

So the screenshot above shows a string of "true" and we tried a bool of True. Trying a bool of true results in an error because of the variable thing:
bool

monstrorivas commented

Looking at the screenshots provided, something looks different in the Tags of the endpoint as compared to the Tags of the model (where the input signals are specified). There is no : in the tag of the endpoint. Can edit the tag from the Azure UI to test?

This how mine looks:
image

And if I try to edit the tag, this is what I have
image

monstrorivas commented

@cherrera2001
Do you have any insights on how to set the Seeq: true tag programmatically with AciWebservice.deploy_configuration? I don't see anything amiss in the steps tried above?

cherrera2001 commented

Looking here deploy_configuration, tags are defined as a dict of type [str,str], so the true should be wrapped in quotes.

insm-cvx commented

Looking here deploy_configuration, tags are defined as a dict of type [str,str], so the true should be wrapped in quotes.

Yep, Seeq="true" is what we tried first (step 4 in my original post). Have you all tested the seeq-azureml functionality with ACI or AKS? I was trying with an ACI endpoint because it was quicker but we're planning to use AKS as the default moving forward.

cherrera2001 commented

Our testing was primarily done with ACI as well as managed endpoints as well. I will do a bit more testing later today.

cherrera2001 commented

I have identified a seemingly new bug with the AzureML REST endpoint that we use to discover endpoints. It returns endpoints with computing types of managed online compute and K8S but not ACI. I am reaching out to the MSFT team to report the issue. In the meantime, a workaround has been identified but is quite complex and would lead to increased latency in launching the add-on. I will wait to hear back on the timeline for the fix from MSFT before we implement the workaround.

insm-cvx commented

Thanks for the update! We'll try it with AKS next week since that's our preferred option anyway.

cherrera2001 commented

I was able to identify a relatively performant workaround and raised a PR.