grothesk/running-open-source-machine-learning-models-on-premises

Material for the talk at ETIM 2024

Running Open-Source Machine-Learning Models On-Premises

Material for the talk at ETIM 2024

Presentation

The slides can be found here.

Demo

Requirements

• to set up the local Kubernetes cluster that is required to run the demo, follow the instructions on https://github.com/grothesk/kserve-on-minikube or https://github.com/deepshore/kserve-locally/tree/docker-desktop - or use a regular cluster and adapt the Makefile accordingly.
• ensure the MinIO client has been installed: https://min.io/docs/minio/linux/reference/minio-mc.html
• install the required Python packages via pipenv:

pipenv sync

• download the model files like this:

make download-model-files 

Steps to perform

For running the demo, perform the following steps:

1. Package model files (MAR) so that torchserveas model serving runtime can work with the model:

make create-mar

2. Create the required directory structure on the object storage and upload the MAR file and the config.properties:

make create-model-buckets upload-model-files

3. Create an InferenceService resource including references to the location of the model files and the model serving runtime to be used:

make deploy-inference-service

4. Prepare the image of the good boy and classify it via HTTP requests:

IMAGE=dog make convert-jpg-to-json classify-picture-of-cute-animal

5. Do the same for another cute animal:

IMAGE=shark make convert-jpg-to-json classify-picture-of-cute-animal