ModelMesh Metrics

ModelMesh Performance Scripts, Dashboard and Pipelines. Follow the instruction to run gRPC inference requests benchmark on a ModelMesh Serving instance on a Kubernetes cluster:

Setup ModelMesh Serving
Create Example Models
Run gRPC Inference Benchmark
Monitoring
Automation with KubeFlow Pipeline

Setup ModelMesh Serving

To quickly stand up a ModelMesh Serving instance on a Kubernetes Cluster, please see Quickstart for detail.

Model Deployment

Switch to the ModelMesh Serving namespace, assuming modelmesh-serving:

kubectl config set-context --current --namespace modelmesh-serving

We have included scripts to deploy models concurrently to a ModelMesh Serving instance. For example, the following deploys 10 simple-string tensorflow models at concurrency of 5:

cd multi_model_tester/
./deployNpredictors.sh 5 simple-string-tf 1 10 deploy_1simple_string_tf_predictor.sh

Verify that the 10 simple-string-tf-* models are Loaded:

kubectl get predictors | grep simple-string-tf

Run Inference Requests

To send requests locally, in a separate terminal run kubectl port-forward svc/modelmesh-serving 8033 and send inference requests to the 10 loaded simple-string-tf-* models using the multi_model_tester.

./multi_model_test -ma "SimpleStringTF" -npm 10 -qps 100 -dur 10

Running the driver from inside the Kubernetes cluster and namespace where the Modelmesh Serving instance is installed:

KUBECTL_COMMAND_HEADERS=false kubectl run -it --rm modelmesh-payload --restart=Never --image=aipipeline/modelmesh-payload:latest --command --namespace modelmesh-serving -- ./multi_model_test -ma 'SimpleStringTF' -u dns:///modelmesh-serving.modelmesh-serving:8033 -npm '10'  -qps '100' -dur '10'

Depending on where the driver is run, the output should be similar to:

QPS: 100
dur: 9.993041819
earliest: 76448
latest: 9990014958
end: 9992985483
reqs: 1000
success: 1.000000
p50: 3.58451ms
p95: 10.701875ms
p99: 29.714907ms
mean: 4.733809ms
max: 37.563672ms

Monitoring

Monitoring ModelMesh Serving metrics using Prometheus and Grafana Dashboard is highly recommended. See Monitoring documentation for details.

Test with KubeFlow Pipeline