GPT-4 Vision for Home Assistant

Image Analyzer for Home Assistant using GPT Vision

🌟 Features · 📖 Resources · ⬇️ Installation · ▶️ Usage · 🪲 How to report Bugs

ha-gpt4vision creates the gpt4vision.image_analyzer service in Home Assistant. This service sends an image to an AI provider and returns the output as a response variable for easy use in automations. Supported providers are OpenAI, LocalAI and Ollama.

Features

Multimodal conversation with AI models
Compatible with OpenAI's API, LocalAI and Ollama
Images can be downscaled for faster processing
Can be installed and updated through HACS and can be set up in the Home Assistant UI

Resources

Check the 📖 wiki for examples on how you can integrate gpt4vision into your Home Assistant or join the 🗨️ discussion in the Home Assistant Community.

Installation

Installation via HACS (recommended)

Search for GPT-4 Vision in Home Assistant Settings/Devices & services
Select wheter you want to use OpenAI or your own LocalAI server for processesing
- For OpenAI's API provide your API key
- For LocalAI enter your IP address and port of your LocalAI server

Manual Installation

Download and copy the gpt4vision folder into your custom_components folder.
Add integration in Home Assistant Settings/Devices & services
Provide your API key or IP address and port of your LocalAI server

Provider specific setup

OpenAI

Simply obtain an API key from OpenAI and enter it in the Home Assistant UI during setup.
A pricing calculator is available here: https://openai.com/api/pricing/.

LocalAI

To use LocalAI, you need to have a LocalAI server running. You can find the installation instructions here. During setup you'll need to provide the IP address of your machine and the port on which LocalAI is running (default is 8000).

Ollama

To use Ollama you first need to install Ollama on your machine. You can download it from here. Once installed you need to run the following command to download the llava model:

ollama run llava

If your Home Assistant is not running on the same computer as Ollama, you need to set the OLLAMA_HOST environment variable.

On Linux:

Edit the systemd service by calling systemctl edit ollama.service. This will open an editor.
For each environment variable, add a line Environment under section [Service]:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Save and close the editor.
Reload systemd and restart Ollama

systemctl daemon-reload
systemctl restart ollama

On Windows:

Quit Ollama from the system tray
Open File Explorer
Right click on This PC and select Properties
Click on Advanced system settings
Select Environment Variables
Under User variables click New
For variable name enter OLLAMA_HOST and for value enter 0.0.0.0
Click OK and start Ollama again from the Start Menu

On macOS:

Open Terminal
Run the following command

launchctl setenv OLLAMA_HOST "0.0.0.0"

Restart Ollama

Service call and usage

After restarting, the gpt4vision.image_analyzer service will be available. You can test it in the developer tools section in home assistant. To get GPT's analysis of a local image, use the following service call.

service: gpt4vision.image_analyzer
data:
  max_tokens: 100
  message: Describe what you see
  image_file: |-
    /config/www/tmp/example.jpg
    /config/www/tmp/example2.jpg
  provider: LocalAI
  model: gpt-4-vision-preview
  target_width: 1280
  temperature: 0.5

The parameters message, max_tokens, image_file, provider and temperature are required. You can send multiple images per service call. Note that each path must be on a new line.

Optionally, the model and target_width properties can be set. For available models check these pages: OpenAI and LocalAI.

How to report a bug or request a feature

Note

Bugs: If you encounter any bugs and have followed the instructions carefully, feel free to file a bug report.
Feature Requests: If you have an idea for a feature, create a feature request.

Create new Issue

jkpe/ha-gpt4vision