/replit-code-v1-3b-truss

Primary LanguagePython

Deploy Replit's Code Completion LM (replit-code-v1-3b) with Baseten

Pre-reqs

  • Set up a Baseten account
  • Clone this repo
  • Install Baseten and Truss client
pip install baseten
pip install truss
  • Auth into Baseten
import baseten
baseten.login("*** BASETEN API KEY ***")

Deploy to Baseten

import baseten
import truss

replitlm_handle = truss.load(".")
baseten.deploy(replitlm_handle, model_name="Replit Code Completion V1 (3B)")

Hardware

This model will deploy to a NVIDIA A10G. Inference time is 3-5 seconds depending on the length of the prompt.