microsoft/azureml-inference-server

Logging traces with "azureml-inference-server-http >= 0.8.0" and "import mlflow"

Opened this issue · 0 comments

Hi,

I am facing a strange behavior with logging traces to App Insights not working when using azureml-inference-server-http 0.8.0 (and higher) and importing mlflow in score.py. This issue does not occur for azureml-inference-server version 0.7.7 and everything works fine.

Environment specs

  • azureml python sdk v1
  • I am using pipenv to avoid dependency issues
  • Tested with Python 3.8, Python 3.10
  • Tested with different versions of mlflow
  • I am using enable_app_insights=True
  • Tested with different versions of azureml-defaults including the latest 1.54.0
  • Deploying to ACI

Use case

I am using the following notebook:
https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.ipynb

Steps to reproduce the issue:

  1. Logging traces works fine if I never import mlflow in the score.py
  2. Logging traces stop working right after I import mlflow.

For example, in the following score.py, "model initialized" is logged to traces, while after importing mlflow, the "second print" never makes it to App Insights traces.

%%writefile score.py
import os
import pickle
import json
import numpy
import joblib
from sklearn.linear_model import Ridge
import time

def init():
    global model
    print ("model initialized" + time.strftime("%H:%M:%S"))
    import mlflow
    print("second print")
    model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'sklearn_regression_model.pkl')
    model = joblib.load(model_path)

def run(raw_data):
...

I am not sure what is happening during runtime that stops traces from being logged.
Any help is much appreciated.