Evaluate LLMs in real time with Street Fighter III

Make LLM fight each other in real time in Street Fighter III.

Which LLM will be the best fighter ?

Our criterias 🔥

They need to be:

Fast: It is a real time game, fast decisions are key
Smart: A good fighter thinks 50 moves ahead
Out of the box thinking: Outsmart your opponent with unexpected moves
Adaptable: Learn from your mistakes and adapt your strategy
Resilient: Keep your RPS high for an entire game

Let the fight begin 🥷

1 VS 1: Mistral 7B vs Mistral 7B

slow-v1.mp4

1 VS 1 X 6 : Mistral 7B vs Mistral 7B

6v6-fast.mp4

1 VS 1: Mistral vs Solar

mistral-small-latest vs solar-1-mini-chat

ms.mp4

mistral-medium-latest vs solar-1-mini-chat

mm.mp4

mistral-large-latest vs solar-1-mini-chat

ml.mp4

A new kind of benchmark ?

Street Fighter III assesses the ability of LLMs to understand their environment and take actions based on a specific context. As opposed to RL models, which blindly take actions based on the reward function, LLMs are fully aware of the context and act accordingly.

Results

Our experimentations (342 fights so far) led to the following leader board. Each LLM has an ELO score based on its results

Ranking

ELO ranking

Model	Rating
🥇openai:gpt-3.5-turbo-0125	1776.11
🥈mistral:mistral-small-latest	1586.16
🥉openai:gpt-4-1106-preview	1584.78
openai:gpt-4	1517.2
openai:gpt-4-turbo-preview	1509.28
openai:gpt-4-0125-preview	1438.92
mistral:mistral-medium-latest	1356.19
mistral:mistral-large-latest	1231.36

Win rate matrix

Explanation

Each player is controlled by an LLM. We send to the LLM a text description of the screen. The LLM decide on the next moves its character will make. The next moves depends on its previous moves, the moves of its opponents, its power and health bars.

Agent based
Multithreading
Real time

Installation

Follow instructions in https://docs.diambra.ai/#installation
Download the ROM and put it in ~/.diambra/roms
Install with pip3 install -r requirements
Create a .env file and fill it with the content like in the .env.example file
Run with make run

Test mode

To disable the LLM calls, set DISABLE_LLM to True in the .env file. It will choose the action randomly.

Positions

x is between 0 and 384
y is between 0 and 224

Logging

Change the logging level in the script.py file.

Credits

Made with ❤️ by the OpenGenerativeAI team @oulianov @Pierre-LouisBJT @Platinn @StanGirard during Mistral Hackathon 2024 in San Francisco

Tokkiu/llm-colosseum