Scikit-Learn Custom Code Sample is Broken

[nnegrey]

Describe the bug
When "Deploy your custom prediction routine" at the step creating the model version. The create fails.

What sample is this bug related to?
https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/notebooks/scikit-learn/custom-prediction-routine-scikit-learn.ipynb

Source code / logs
Running inside colab: ERROR: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error: "Failed to load model: Unexpected error when loading the model: 0 (Error code: 0)"

System Information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CoLab
• Framework and version (Tensorflow, scikit-learn, XGBoost): scikit-learn
• Python version: Python 3.6.7
• Exact command to reproduce:

gcloud beta ai-platform versions create $VERSION_NAME \
  --model $MODEL_NAME \
  --runtime-version 1.13 \
  --python-version 3.5 \
  --origin gs://$BUCKET_NAME/custom_prediction_routine_tutorial/model/ \
  --package-uris gs://$BUCKET_NAME/custom_prediction_routine_tutorial/my_custom_code-0.1.tar.gz \
  --prediction-class predictor.MyPredictor

[alecglassford]

I was able to reproduce this in Colab and I think I found the issue:

Colab has (and the library installation step doesn't override) the latest version of scikit-learn installed (0.21.1). But AI Platform runtime version 1.13 has scikit-learn 0.20.2, which vendors a different version of joblib. I think the problem is that scikit-learn 0.20.2's joblib (on the prediction node) tries to load a model exported by scikit-learn 0.21.1's joblib (in Colab).

Replacing the following installation line:

! pip install numpy scikit-learn

with

! pip install numpy scikit-learn==0.20.2

(and then restarting the runtime) works as as a fix.