SteliosGian/mlserver-deployment

A deployment using Seldon's open source MLServer

MLServer Deployment

Deployment of a Fake news detector using Seldon's MLServer open source package.

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
3. MLServer

About The Project

A simple fake news detector model that detects if a text is considered as fake or not. To accomplish this, we used the Bert pretrained model to make inference based on our new dataset.
The goal is to create an inference server based on Seldon's open source package MLServer, and generate predictions based on a text input.

Built With

• Docker
• MLServer
• Pipenv
• PyTorch

Dataset

The dataset used in this project is the "REAL and FAKE news dataset" which is taken from Kaggle. The dataset contains 3 columns. The "title" column, the "text" column and the "label" column which is the target label.

Getting Started

Before the inference server is created, a trained model needs to exist in the src/inference/model directory. The model needs to be named model.pt as indicated in the model-setttings.json file.

The src/fake_news_detector/train.py file can be used to train a new model and the Kaggle dataset needs to be placed inside the src/data directory.

The run-local.sh script takes one argument, depending on whether you want to do training or inference. The default argument is inference.

To run the training phase, type:

bash run-local.sh training

After that, a PyTorch model names model.pt is placed inside the src/inference/temp_model directory. Copy that model in the src/inference/model directory and run the inference phase to start the server that will use that model.

Otherwise, just type:

bash run-local.sh

This script installs all the necessary packages from Pipfile.lock and creates the server using the MLServer package.

The inference can be accessed from:

http://<URL>/v2/models/<MODEL_NAME/infer

Which in our case, since we deploy it locally, is:

http://0.0.0.0:8080/v2/models/fake-news-detector/infer

An example payload is:

{
  "inputs": [
    {
      "name": "text",
      "shape": [1],
      "datatype": "BYTES",
      "data": "This is a fake text"
    }
  ]
}

An example using curl:

curl -X 'POST' \
  'http://0.0.0.0:8080/v2/models/fake-news-detector/infer' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "inputs": [
    {
      "name": "text",
      "shape": [1],
      "datatype": "BYTES",
      "data": "This is a fake text"
    }
  ]
}'

MLServer

MLServer is a way to serve ML models through a REST and gRPC interface.
As mentioned in the official documentation, MLServer supports:

• Multi-model serving
• Parallel inference
• Adaptive batching
• Scalability with Kubernetes native frameworks, such as Seldon Core and KServe
• Standard V2 Inference Protocol on REST and gRPC

Prerequisites

Docker must be installed in order to create the server.