testing with DOTA dataset

[geobao]

Hi, thanks for your great work!

I split the testing with the default configuration split_configs/dota1_0/ss_test.json

Then I run the test as below:

EVAL_METRICS=mAP && \ CONFIG_FILE=configs/obb/oriented_rcnn/faster_rcnn_orpn_r50_fpn_1x_dota10.py && \ CHECKPOINT_FILE=ckpt/faster_rcnn_orpn_r50_fpn_1x_dota10_epoch12.pth && \ python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} --eval ${EVAL_METRICS}

Everything runs fine. However I see that all AP scores are 0 because it does not pick up ground truth.

`root@673e18a920bd:/work/OBBDetection# EVAL_METRICS=mAP && CONFIG_FILE=configs/obb/oriented_rcnn/faster_rcnn_orpn_r50_fpn_1x_dota10.py && CHECKPOINT_FILE=ckpt/faster_rcnn_orpn_r50_fpn_1x_dota10_epoch12.pth && python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} --eval ${EVAL_METRICS}
load checkpoint from local path: ckpt/faster_rcnn_orpn_r50_fpn_1x_dota10_epoch12.pth
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 10833/10833, 4.2 task/s, elapsed: 2557s, ETA: 0s
Merging patch bboxes into full image!!!
Multiple processing
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 937/937, 39.0 task/s, elapsed: 24s, ETA: 0s
Used time: 26.0 s

Start calculate mAP!!!
Result is Only for reference, final result is subject to DOTA_devkit

+--------------------+-----+-------+--------+--------+
| class | gts | dets | recall | ap |
+--------------------+-----+-------+--------+--------+
| large-vehicle | 0 | 23163 | 0.0000 | 0.0000 |
| swimming-pool | 0 | 2190 | 0.0000 | 0.0000 |
| helicopter | 0 | 439 | 0.0000 | 0.0000 |
| bridge | 0 | 5914 | 0.0000 | 0.0000 |
| plane | 0 | 4905 | 0.0000 | 0.0000 |
| ship | 0 | 21624 | 0.0000 | 0.0000 |
| soccer-ball-field | 0 | 1239 | 0.0000 | 0.0000 |
| basketball-court | 0 | 784 | 0.0000 | 0.0000 |
| ground-track-field | 0 | 901 | 0.0000 | 0.0000 |
| small-vehicle | 0 | 37442 | 0.0000 | 0.0000 |
| baseball-diamond | 0 | 1287 | 0.0000 | 0.0000 |
| tennis-court | 0 | 2005 | 0.0000 | 0.0000 |
| roundabout | 0 | 1076 | 0.0000 | 0.0000 |
| storage-tank | 0 | 5646 | 0.0000 | 0.0000 |
| harbor | 0 | 6926 | 0.0000 | 0.0000 |
+--------------------+-----+-------+--------+--------+
| mAP | | | | 0.0000 |
+--------------------+-----+-------+--------+--------+
`

Do I have to reformat the groundtruth of dota dataset before splitting?
The configuration file points to the test partition that has been splitted, so everything is running by default. Am I missing something?

Thanks again

[jbwang1997]

ss_test.json doesn't specify ann_dir, so the dataset doesn't have ground-truth information. You can use ss_val.json to split your test set.

[geobao]

Thanks!

[fhaji75]

@geobao
Can you solve this problem?
If we use ss_val.json , we need labelTxt folder for test images.

[Windy4096]

DOTA does not provide label for test dataset
Only the label of train and val dataset can we access