sshh12/multi_token

Summarize video

Closed this issue · 1 comments

linchen111 commented

If my multi is continuous images from like screenshots, what should be my prompt when I use Mistral-7B-LoRA-Multi-VisionCLIPPool-LLAVA

sshh12 commented

This is the format I used:

multi_token/scripts/llava_gpt_build_multi_image_finetune_dataset.py

Line 88 in 6eb9813

if random.choice([True, False]): 

<image><image><image> What is happening in these frames?

Although not sure how well it work given my training data was mainly compare/contrast rather than video understanding.

It's only trained for up to 6 images, may work for more though.