joshbickett/multimodal-gamer

A framework to enable multimodal models to play games on a computer.

Multimodal Gamer

A framework to enable multimodal models to play games on a computer.

Using the same inputs and outputs as a human operator, the model plays a game!

Key Features

• Compatibility: Designed to be compatible with various multimodal models and games
• Models: Currently integrated with GPT-4
• Supported Games:
  • Super Mario 64: https://www.smbgames.be/super-mario-64.php
  • Poker: https://www.247freepoker.com/
  • Chess: https://www.chess.com/
    • Requires Chess.com Keyboard: https://chromewebstore.google.com/detail/chesscom-keyboard/bghaancnengidpcefpkbbppinjmfnlhh

Demo of Multimodal Gamer playing on Chess.com

readme.mp4

Quick install

1. Create a virtual environment:

   python3 -m venv env

2. Activate the virtual environment (Mac/Linux):

   source env/bin/activate

3. Install the required packages:

   pip install -r requirements.txt

4. Add your OpenAI API key:

   export OPENAI_API_KEY=yourkeyhere

Running the Game

To run a specific game, use the -game flag followed by the name of the game. For example, to play chess:

python main.py --game chess