rzig/modelrocket

Self-service ML model serving for HackIllinois 2023

Ben Guan, Vijay Shah, Jacob Stolker, Ryan Ziegler

Overview

ModelRocket was designed from the ground up to be horizontally scalable, and is divided into n core services:

1. Coordinator: responsible for autoscaling and processing updates to authentication. Written in Python, interfaces with Redis.
2. Model Manager: responsible for processing new model uploads and getting them ready for deployment. Written in NodeJS, interfaces with the Coordinator and Redis.
3. Inference Gateway: responsible for routing inference requests for a model to an appropriate upstream server. Written in Python, interfaces with Redis.
4. Inference Host: custom software running on servers to manage active models and provide fast inference. Written in Python, interfaces with Redis.
5. Python Client: allows the user to upload their model to our service with a single line of code. Detects model type and converts it to ONNX, then makes the necessary post requests to store the model in MongoDB and S3.
6. Web UI: allows users to manage their models and update API keys. Written in React.

Client Setup

• Pushing model to our service
  • usage: modelrocket.upload(model, model_name, input_type, input_shape)
    • This will output an api token and a model uuid.
• Making an inference request
  • usage: make a post request to the following link http://127.0.0.1:5000/inference with the following JSON body:

    {
        "token": api_token, 
        "model": model_uuid, 
        "inputs": {"modelInput": your_input_to_the_model}
    }
    

  • This will output a JSON object with the results of your inference.
• Website
  • Gives demonstration of how to write code for our service

Server Setup (required only if you want to host)

1. Install necessary packages for each service

   • Coordinator:
     • run cd coordinator
     • run pipenv install
     • run cd ..
   • Model Manager:
     • cd model-manager
     • Add a .env with the following variables:

         MONGODB_URL=
         ACCESS_KEY_ID=
         SECRET_ACCESS_ID=
         

     • run npm install
   • Inference Gateway:
     • run cd coordinator
     • run pipenv install
     • run cd ..
   • Inference Host:
     • run cd coordinator
     • run pipenv install
     • run cd ..

2. Run docker-compose up --detach

3. Run the following commands to set up environment variables:

   • AWS_ACCESS_KEY_ID=<your_access_key_id> AWS_SECRET_ACCESS_KEY=<your_secret_access_key></your_secret_access_key> S3_ENDPOINT=http://localhost:9000/ PORT=5002 python host.py
   • REDIS_HOST=localhost REDIS_PORT=6379 REDIS_PASSWORD=<your_password> REDIS_DB=0 python -m flask --app gateway run
   • REDIS_HOST=localhost REDIS_PORT=6379 REDIS_PASSWORD=<your_password> REDIS_DB=0 python assigner.py

4. S3 setup:

   • Go to localhost:9000
   • Sign in with your access key-id and secret-key.
   • Create a new bucket called models