jacobmarks/fiftyone-image-captioning-plugin

Caption images across your datasets with state of the art models from Hugging Face and Replicate!

Image Captioning Plugin

Plugin Overview

This plugin lets you generate and store captions for your samples using state-of-the-art image captioning models.

Supported Models

This version of the plugin supports the following models:

• BLIP Base from Hugging Face
• BLIPv2 (via Replicate)
• Fuyu-8b from Adept AI (via Replicate)
• GiT from Hugging Face
• Llava-1.5-7b from Hugging Face
• Llava-13b (via Replicate)
• Qwen-vl-chat (via Replicate
• ViT-GPT2 from Hugging Face

Feel free to fork this plugin and add support for other models!

Installation

Pre-requisites

1. If you plan to use it, install the Hugging Face transformers library:

pip install transformers

2. If you plan to use it, install the Replicate library:

pip install replicate

And add your Replicate API key to your environment:

export REPLICATE_API_TOKEN=<your-api-token>

Install the plugin

fiftyone plugins download https://github.com/jacobmarks/fiftyone-image-captioning-plugin

Operators

caption_images

• Applies the selected image captioning model to the desired target view, and stores the resulting captions in the specified field on the samples.