openplayground

An LLM playground.

Forked from https://github.com/nat/openplayground, I've added more default models and some cosmetic updates. Feel free to submit PR's.

Features

Use any model from OpenAI, Anthropic, Cohere, Forefront, HuggingFace, Aleph Alpha, Replicate, Banana and llama.cpp.
Full playground UI, including history, parameter tuning, keyboard shortcuts, and logprops.
Compare models side-by-side with the same prompt, individually tune model parameters, and retry with different parameters.
Automatically detects local models in your HuggingFace cache, and lets you install new ones.
Works OK on your phone.
Probably won't kill everyone.

How to run for development

# set up python env
python -m venv venv
source venv/bin/activate

# install
make install

# in one terminal run the frontend
make frontend

# in another terminal
make backend
# or
make backend-dev

# clean config (after making changes to model.json)
make clean

Contributions welcome. Some ideas:

Add a token counter #1
Ability to Delete local Hugging Face models from the cache #2
Display progress-bar when downloading Hugging Face models #3
Add a cost counter compare page #4
Measure and display time to first token #5
Chatgpt plugin support #6

Adding models to openplayground

Models and providers have three types in openplayground:

Searchable
Local inference
API

You can add models in server/models.json with the following schema:

Local inference

For models running locally on your device you can add them to openplayground like the following (a minimal example):

"llama": {
    "api_key" : false,
    "models" : {
        "llama-70b": {
            "parameters": {
                "temperature": {
                    "value": 0.5,
                    "range": [
                        0.1,
                        1.0
                    ]
                },
            }
        }
    }
}

Keep in mind you will need to add a generation method for your model in server/app.py. Take a look at local_text_generation() as an example.

API Provider Inference

This is for model providers like OpenAI, cohere, forefront, and more. You can connect them easily into openplayground (a minimal example):

"cohere": {
    "api_key" : true,
    "models" : {
        "xlarge": {
            "parameters": {
                "temperature": {
                    "value": 0.5,
                    "range": [
                        0.1,
                        1.0
                    ]
                },
            }
        }
    }
}

You will need to add a generation method for your model in server/app.py. Take a look at openai_text_generation() or cohere_text_generation() as an example.

Searchable models

We use this for Huggingface Remote Inference models, the search endpoint is useful for scaling to N models in the settings page.

"provider_name": {
    "api_key": true,
    "search": {
        "endpoint": "ENDPOINT_URL"
    },
    "parameters": {
        "parameter": {
            "value": 1.0,
            "range": [
                0.1,
                1.0
            ]
        },
    }
}

Credits

Instigated by Nat Friedman. Initial implementation by Zain Huda as a repl.it bounty. Many features and extensive refactoring by Alex Lourenco.

quilime/openplayground