facebookresearch/chameleon

Poor quality in responses and image recognition

apresence opened this issue · 1 comments

I'm using the 7b model on Ubuntu 22.04 running a RTX 4090 FE, Cuda 12.2 drivers, and I used the docker compose yaml. I also have an RTX 8000 in the system, but I have configured Docker to only enable the 4090 in the container:

          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [ gpu ]

Image recognition seems poor. I uploaded a picture of a MacBook and first it said it was an Acer running Windows, then it said it was Ubuntu with a welcome screen.

It will often reply "I'm sorry, but I'm unable to comply with that request" or similar to innocuous requests such as "describe this image" (of a cat).

Also, sometimes it will start repeating itself over and over until the token limit has been reached. I've attached a screenshot for reference.

image

OS/Config info:

# uname -a
Linux tau 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
# nvidia-smi
Wed Jun 26 03:45:59 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        On  | 00000000:01:00.0 Off |                  Off |
|  0%   40C    P5              69W / 450W |  14679MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Quadro RTX 8000                On  | 00000000:04:00.0 Off |                  Off |
| 33%   33C    P8              12W / 260W |      3MiB / 49152MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     61418      C   python                                    14128MiB |
|    0   N/A  N/A     61978      C   /usr/bin/python3                            540MiB |
+---------------------------------------------------------------------------------------+

Let me know if there are additional details I can provide. I'm also happy to test new code/settings or provide logs if it will be helpful.

This is not unexpected behavior. Asking less open ended questions about passed images should help.