open-mmlab/mmdetection

training grounding dino with 139 classes

sheethalb opened this issue · 2 comments

sheethalb commented

Thanks for your error report and we appreciate it a lot.

Checklist

  1. I have searched related issues but cannot get the expected help.
  2. I have read the FAQ documentation but cannot get the expected help.
  3. The bug has not been fixed in the latest version.

Describe the bug
I am training grounding dino model with 139 classes. The mAP per class is close to 0 even after 50 epochs. When i train on random 80 classes from the list, I get much higher average mAP_50 of 70%.

Reproduction

  1. What command or script did you run?
    PYTHONPATH="$(dirname $0)/..":$PYTHONPATH
    torchrun
    --nnodes=$NNODES
    --node_rank=$NODE_RANK
    --master_addr=$MASTER_ADDR
    --nproc_per_node=$GPUS
    --master_port=$PORT
    ./tools/train.py
    $CONFIG
    --launcher pytorch ${@:3}

with the gdino swinb config file
all other settings were default, except for the class names and the num_classes parameter.

  1. Did you make any modifications on the code or config? Did you understand what you have modified?
    only the num_classes=141 and the max_text_len=1024 as we have 139 classes.

  2. What dataset did you use?
    https://www.sciencedirect.com/science/article/abs/pii/S095741742401443X
    its an dataset of natural image categories like flower, bus, etc of objects in paintings.

Environment

  1. Please run python mmdet/utils/collect_env.py to collect necessary environment information and paste it here.
    NA
  2. You may add addition that may be helpful for locating the problem, such as

I modified the max_text_length for the contrastive_config = 1024 and the bbox_head (num_classes=141)
neither helped resolve the issue. I also modified the max_num_entities in glip.py to be 1024
def create_positive_map(tokenized,
tokens_positive: list,
max_num_entities: int = 1024) -> Tensor:

Is there anything in the pretraining scheme that limits the max number of classes to the coco number 80 ?

sheethalb commented

i trained it with a different weight decay, since my dataset is quite small. and now after the above changes, only the first 87 classes achieve a reasonable mAP of 0.20 (within first 15 epochs). the rest are all 0. if i train on the last 80 classes, it achieves a reasonable mAP as well (as high as 0.8 for some classes).

the loss continues to decrease but the class wise metrics are like this -

+--------------------+-------+--------+--------+-------+-------+-------+
| category | mAP | mAP_50 | mAP_75 | mAP_s | mAP_m | mAP_l |
+--------------------+-------+--------+--------+-------+-------+-------+
| ant | 0.018 | 0.02 | 0.02 | nan | 0.036 | nan |
| camel | 0.089 | 0.119 | 0.098 | nan | 0.066 | 0.175 |
| jewellery | 0.027 | 0.035 | 0.035 | 0.011 | 0.059 | 0.0 |
| frog | 0.107 | 0.174 | 0.146 | nan | 0.0 | 0.241 |
| physalis | 0.0 | 0.0 | 0.0 | nan | 0.0 | nan |
| celery | 0.205 | 0.383 | 0.083 | nan | 0.025 | 0.325 |
| cauliflower | 0.169 | 0.558 | 0.147 | nan | nan | 0.24 |
| pepper | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| ranunculus | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| chess flower | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| cigarette | 0.001 | 0.001 | 0.001 | 0.001 | nan | nan |
| matthiola | 0.0 | 0.0 | 0.0 | nan | 0.0 | nan |
| cabbage | 0.104 | 0.148 | 0.137 | nan | 0.303 | 0.03 |
| earring | 0.192 | 0.395 | 0.089 | 0.275 | 0.301 | nan |
| dandelion | 0.0 | 0.0 | 0.0 | nan | 0.0 | 0.0 |
| neroli | 0.0 | 0.0 | 0.0 | nan | 0.0 | nan |
| dragonfly | 0.541 | 0.771 | 0.771 | nan | 0.65 | 0.569 |
| hyacinth | 0.0 | 0.0 | 0.0 | nan | 0.0 | 0.0 |
| reptile/amphibia | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| apricot | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| snake | 0.292 | 0.364 | 0.364 | 0.0 | nan | 0.454 |
| lizard | 0.3 | 0.423 | 0.423 | nan | 0.477 | 0.3 |
| asparagus | 0.019 | 0.034 | 0.028 | nan | 0.059 | nan |
| spring onion | 0.13 | 0.265 | 0.049 | nan | 0.253 | 0.019 |
| snowflake | 0.215 | 0.505 | 0.0 | 0.425 | 0.0 | nan |
| moth | 0.138 | 0.231 | 0.224 | 0.081 | 0.002 | 0.36 |
| poppy | 0.018 | 0.027 | 0.015 | nan | 0.024 | nan |
| columbine | 0.003 | 0.006 | 0.0 | 0.0 | 0.006 | nan |
| rabbit | 0.307 | 0.343 | 0.343 | nan | 0.01 | 0.589 |
| geranium | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| crab | 0.296 | 0.556 | 0.151 | nan | 0.115 | 0.459 |
| radish | 0.007 | 0.036 | 0.0 | 0.0 | 0.051 | 0.0 |
| big cat | 0.482 | 0.617 | 0.44 | nan | 0.516 | 0.473 |
| jan steen jug | 0.632 | 0.791 | 0.791 | nan | 0.783 | 0.715 |
| monkey | 0.28 | 0.382 | 0.32 | nan | 0.7 | 0.249 |
| snail | 0.055 | 0.244 | 0.016 | 0.206 | 0.035 | 0.0 |
| bellflower | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| lilac | 0.001 | 0.002 | 0.002 | 0.0 | 0.007 | nan |
| pot | 0.053 | 0.062 | 0.062 | 0.7 | 0.047 | 0.098 |
| peony | 0.083 | 0.103 | 0.103 | nan | 0.143 | 0.082 |
| coffeepot | 0.025 | 0.029 | 0.029 | nan | 0.022 | 0.063 |
| hazelnut | 0.001 | 0.001 | 0.001 | 0.0 | 0.007 | nan |
| censer | 0.337 | 0.45 | 0.435 | nan | 0.336 | 0.512 |
| artichoke | 0.248 | 0.402 | 0.353 | nan | 0.284 | 0.308 |
| dahlia | 0.145 | 0.152 | 0.146 | 0.0 | 0.006 | 0.52 |
| sniffing | 0.02 | 0.093 | 0.0 | 0.0 | 0.059 | 0.017 |
| fly | 0.151 | 0.197 | 0.183 | 0.05 | 0.212 | 0.141 |
| deer | 0.027 | 0.039 | 0.032 | 0.037 | 0.025 | 0.098 |
| caterpillar | 0.491 | 0.663 | 0.484 | 0.0 | 0.57 | 0.608 |
| garlic | 0.099 | 0.205 | 0.071 | 0.168 | 0.159 | nan |
| blackberry | 0.028 | 0.094 | 0.002 | 0.023 | 0.104 | nan |
| chalice | 0.252 | 0.331 | 0.229 | nan | 0.083 | 0.503 |
| lobster | 0.428 | 0.534 | 0.534 | nan | 0.0 | 0.637 |
| necklace | 0.349 | 0.47 | 0.361 | 0.305 | 0.464 | 0.43 |
| bug | 0.221 | 0.321 | 0.278 | 0.259 | 0.321 | 0.058 |
| insect | 0.156 | 0.209 | 0.204 | 0.05 | 0.278 | 0.023 |
| prawn | 0.231 | 0.307 | 0.265 | nan | 0.267 | 0.088 |
| bracelet | 0.347 | 0.477 | 0.374 | 0.336 | 0.487 | nan |
| carrot | 0.075 | 0.194 | 0.042 | 0.008 | 0.105 | 0.7 |
| cornflower | 0.022 | 0.037 | 0.022 | 0.01 | 0.073 | 0.9 |
| pumpkin | 0.192 | 0.292 | 0.204 | nan | 0.177 | 0.217 |
| orange | 0.239 | 0.364 | 0.2 | 0.149 | 0.266 | 0.438 |
| walnut | 0.065 | 0.082 | 0.067 | 0.147 | 0.138 | 0.148 |
| cat | 0.621 | 0.823 | 0.817 | nan | 0.327 | 0.764 |
| daisy | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| forget-me-not | 0.231 | 0.497 | 0.126 | 0.233 | 0.1 | nan |
| carafe | 0.123 | 0.215 | 0.103 | nan | 0.11 | 0.276 |
| match | 0.098 | 0.158 | 0.08 | 0.081 | 0.171 | nan |
| beer stein | 0.628 | 0.752 | 0.739 | 0.691 | 0.624 | 0.597 |
| tobacco-box | 0.425 | 0.658 | 0.498 | 0.162 | 0.683 | 0.402 |
| violet | 0.004 | 0.005 | 0.005 | 0.0 | 0.01 | nan |
| pomander | 0.518 | 0.823 | 0.67 | 0.556 | 0.581 | nan |
| bottle | 0.351 | 0.485 | 0.366 | 0.25 | 0.382 | 0.406 |
| candle | 0.448 | 0.698 | 0.481 | 0.132 | 0.8 | 0.593 |
| heliotrope | 0.04 | 0.103 | 0.02 | 0.05 | 0.034 | nan |
| wine bottle | 0.238 | 0.318 | 0.252 | 0.034 | 0.185 | 0.573 |
| strawberry | 0.175 | 0.281 | 0.174 | 0.144 | 0.168 | 0.483 |
| pomegranate | 0.19 | 0.294 | 0.191 | 0.0 | 0.219 | 0.49 |
| whale | 0.312 | 0.522 | 0.334 | 0.326 | 0.313 | 0.45 |
| lily of the valley | 0.466 | 0.835 | 0.484 | 0.468 | nan | nan |
| iris | 0.128 | 0.141 | 0.136 | 0.003 | 0.004 | 0.601 |
| tobacco | 0.386 | 0.657 | 0.394 | 0.314 | 0.605 | nan |
| olive | 0.04 | 0.111 | 0.013 | 0.029 | nan | 0.315 |
| tobacco-packaging | 0.6 | 0.715 | 0.651 | 0.506 | 0.621 | 0.9 |
| meat | 0.187 | 0.264 | 0.223 | 0.0 | 0.05 | 0.384 |
| daffodil | 0.112 | 0.201 | 0.116 | 0.0 | 0.366 | 0.122 |
| melon | 0.12 | 0.208 | 0.149 | 0.0 | 0.011 | 0.365 |
| fire | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| petunia | 0.0 | 0.0 | 0.0 | nan | 0.0 | nan |
| mushroom | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| teapot | 0.0 | 0.0 | 0.0 | nan | 0.0 | 0.0 |
| ring | 0.0 | 0.0 | 0.0 | 0.0 | nan | nan |
| pig | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| ashtray | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| cheese | 0.0 | 0.0 | 0.0 | nan | 0.0 | 0.0 |
| onion | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| cup | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| nut | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| fig | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| drinking vessel | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| donkey | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| holding the nose | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| lily | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| smoke | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| bread | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| currant | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| glass without stem | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| anemone | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| mammal | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| chimney | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| smoking equipment | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| bivalve | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| butterfly | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| gloves | 0.0 | 0.0 | 0.0 | nan | 0.0 | 0.0 |
| lemon | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| horse | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| plum | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| jasmine | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | nan |
| pear | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| glass with stem | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| vegetable | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| carnation | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| jug | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| goat | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| fish | 0.0 | 0.0 | 0.0 | nan | 0.0 | 0.0 |
| apple | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| tulip | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| cherry | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| cow | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| animal corpse | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| dog | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| fruit | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| bird | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| rose | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| peach | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| sheep | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| pipe | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| grapes | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| flower | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
+--------------------+-------+--------+--------+-------+-------+-------+

sheethalb commented

i had to change the max_text_len in bert.py in the language model
its under mmdet/models/language_model/bert.py

thank you