can't get the good results on the FS datasets

Question

can't get the good results on the FS datasets

StarWanan opened this issue 2 years ago · 10 comments

Hi!

I tried to run the test.py , but I can't get the good results on the FS dataset. Here are the results I got:

[pebal][INFO] validating cityscapes dataset ... 
 labeled: 1966976, correct: 1933282: 100%|█████████████████████████████████████████████████████████████| 500/500 [33:56<00:00,  4.07s/it]
[pebal][CRITICAL] current mIoU is 0.895736885447597, mAcc is 0.979122587176078 

100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [02:17<00:00,  1.37s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_ls: 0.5020263181424963                                                                                    
[pebal][CRITICAL] AUPRC score for Fishyscapes_ls: 0.0025106950112928753 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_ls: 0.7372275091455253 

[pebal][INFO] validating Fishyscapes_static dataset ... 
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:39<00:00,  1.31s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_static: 0.4023022576107206 
[pebal][CRITICAL] AUPRC score for Fishyscapes_static: 0.012901811279743608 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_static: 0.8040210492683544

The FS datasets is downloaded from the link you provided (synboost GitHub), and the corresponding folder structure is:

fishyscapes/
├── LostAndFound
│ ├── entropy
│ ├── labels
│ ├── labels_with_ROI
│ ├── logit_distance
│ ├── mae_features
│ ├── original
│ ├── semantic
│ └── synthesis
└── Static
     ├── entropy
     ├── labels
     ├── labels_with_ROI
     ├── logit_distance
     ├── mae_features
     ├── original
     └── semantic

In Static and LostAndFound, I use the images in the labels folder and the original folder.

Here is the code in my modified data_loader.py:

def __init__(self, split='Static', root="", transform=None):
    """Load all filenames."""
    self.transform = transform
    self.root = root
    self.split = split  # ['Static', 'LostAndFound']
    self.images = []  # list of all raw input images
    self.targets = []  # list of all ground truth TrainIds images
    # filenames = os.listdir(os.path.join(root, self.split, 'images'))    
    filenames = os.listdir(os.path.join(root, self.split, 'original'))    # final folder
    root = os.path.join(root, self.split)
    for filename in filenames:
        if os.path.splitext(filename)[1] == '.png':
            f_name = os.path.splitext(filename)[0]
            # ======= old ======= #
            # filename_base_img = os.path.join("images", f_name)
            # filename_base_labels = os.path.join("labels", f_name.replace("leftImg8bit", "labels"))
            # ======= final ======= #
            filename_base_img = os.path.join("original", f_name)
            filename_base_labels = os.path.join("labels", f_name)
            # ========================== #
            self.images.append(os.path.join(root, filename_base_img + '.png'))
            self.targets.append(os.path.join(root, filename_base_labels + '.png'))
            self.images = sorted(self.images)
            self.targets = sorted(self.targets)

Thanks! Looking forward to your reply.

Answer 1 · 2022-08-14T04:27:16.000Z

Do you perform the pre-process, which normalise the image in the certain range, in this line

Answer 2 · 2022-08-14T08:25:05.000Z

I didn't use the normalize function in the data_loader.py you mentioned, because I noticed these lines in test.py:

transform = Compose([ToTensor(), Normalize(config.image_mean, config.image_std)])

fishyscapes_ls = Fishyscapes(split='LostAndFound', root=config.fishy_root_path, transform=transform)
fishyscapes_static = Fishyscapes(split='Static', root=config.fishy_root_path, transform=transform)

And I think torchvision.transforms.ToTensor will normalise the image in the certain range（[0,1]）:

def __getitem__(self, i):
    """Return raw image, trainIds as torch.Tensor or PIL Image"""
    image = Image.open(self.images[i]).convert('RGB')
    target = Image.open(self.targets[i]).convert('L')

    if self.transform is not None:
        image, target = self.transform(image, target)
        
    return image, target

Answer 3 · 2022-08-14T08:36:19.000Z

Could you please upload some visualisations here?
For example, 1). input image, 2). prediction panel, 3). ood_score panel and 4). pixel-wise label.

Please check that the visualisation for the ood_score should be similar as what I've shown in this issue.

Answer 4 · 2022-08-14T09:20:07.000Z

input image：

anomaly_score:

I got the result using code like below but I am not sure if it is correct

# test.py
from utils.wandb_upload import *
visual_tool = Tensorboard(config=config)

valid_anomaly(model=model, epoch=0, test_set=fishyscapes_static,
                  data_name='Fishyscapes_static', my_wandb=visual_tool, logger=logger)

# valid.py
if my_wandb is not None:
    my_wandb.upload_wandb_info(current_step=epoch, info_dict=curr_info)
    my_wandb.upload_ood_image(current_step=epoch, energy_map=focus_area, reserve_map=None,
                              img_number=upload_img_num, data_name=data_name)

pixel-wise label:

And I'm working on how to get prediction panel。

Answer 5 · 2022-08-14T10:28:21.000Z

Hi @StarWanan , may I ask what have you modified for the code, because our data structure should be totally the same as the synboost.

I downloaded the ckpt & the repository, and the test.py can work without any modification.

Our fishyscape structure is :

fishyscapes
├── LostAndFound
│   ├── entropy
│   ├── labels
│   ├── labels_with_ROI
│   ├── logit_distance
│   ├── mae_features
│   ├── original
│   ├── semantic
│   └── synthesis
└── Static
    ├── entropy
    ├── labels
    ├── labels_with_ROI
    ├── logit_distance
    ├── mae_features
    ├── original
    └── semantic

The testing log is shown below:

/home/yuyuan/anaconda3/envs/pebal/bin/python /home/yuyuan/work_space/PEBAL/code/test.py
[pebal][CRITICAL] restoring pebal ckpt from /home/yuyuan/work_space/PEBAL/ckpts/pretrained_ckpts/best_ad_ckpt.pth 
[pebal][INFO] validating Fishyscapes_ls dataset ... 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:43<00:00,  1.03s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_ls: 0.9896340042387362 
[pebal][CRITICAL] AUPRC score for Fishyscapes_ls: 0.588127504166506 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_ls: 0.04766274243599872 
[pebal][INFO] validating Fishyscapes_static dataset ... 
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:30<00:00,  1.03s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_static: 0.9960209766813635 
[pebal][CRITICAL] AUPRC score for Fishyscapes_static: 0.9208418393359893 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_static: 0.015202855528294194 

Process finished with exit code 0

Answer 6 · 2022-08-14T18:29:06.000Z

Thank you for your reply。

The code I got about a month ago is not the same as it is now. and I tried the latest code on Github, still no change in result.

The testing log is shown below:

[pebal][CRITICAL] restoring pebal ckpt from /home/zyx/PEBAL/ckpts/pebal/best_ad_ckpt.pth 
[pebal][INFO] validating Fishyscapes_static dataset ... 
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:38<00:00,  1.28s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_static: 0.4023022576107206 
[pebal][CRITICAL] AUPRC score for Fishyscapes_static: 0.012901811279743608 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_static: 0.8040210492683544

And I noticed that the path to load the .pth file is different, is there a difference between the code you run and the one on github?

C.pebal_weight_path = os.path.join(C.root_dir, 'ckpts', 'pebal', 'best_ad_ckpt.pth')
C.pretrained_weight_path = os.path.join(C.root_dir, 'ckpts', 'pretrained_ckpts', 'cityscapes_best.pth')

Answer 7 · 2022-08-15T02:19:08.000Z

Nope, it is totally the same as the supported checkpoint, hahaha. I just put the checkpoint in a different directory, and it will not cause any difference in the results.

1). You can view the commit history that the "test.py" hasn't been changed for 2 months. I cannot understand what you mean by modifying the code.

2). What you have for the result makes non-sense, as you can even have a lot better results for the started checkpoint (the Nvidia one).

Here is the log:

[pebal][INFO] Load checkpoint from file /home/yuyuan/work_space/PEBAL/ckpts/pretrained_ckpts/cityscapes_best.pth, Time usage:
	IO: 0.6330766677856445, restore snapshot: 0.011519193649291992 
[pebal][INFO] validating Fishyscapes_ls dataset ... 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:46<00:00,  1.07s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_ls: 0.9371502892755115 
[pebal][CRITICAL] AUPRC score for Fishyscapes_ls: 0.16046114663246977 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_ls: 0.41779574363621813 
[pebal][INFO] validating Fishyscapes_static dataset ... 
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:30<00:00,  1.02s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_static: 0.9590370905998604 
[pebal][CRITICAL] AUPRC score for Fishyscapes_static: 0.4168195788594434 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_static: 0.17780031807571348

Please start to work from that one to debug your code. I don't think any cityscape pretrained checkpoints can have such a low AUPRC (e.g., 0.012901811279743608), it is impossible.

3). I have updated the wandb log in here. It includes the code (test.py file), the visualisations (some of the them are visually very different by compared with yours) and the final results (for both of the started nvidia ckpt and our ckpt).

Answer 8 · 2022-08-15T02:23:34.000Z

I'm closing this issue, as I don't want to waste more time on this question.

Answer 9 · 2023-02-08T20:07:00.000Z

I also had this issue, but I couldn't figure out how to solve it. I think it has to do with loading the checkpoint but not sure. I gave up in the end.

Answer 10 · 2023-02-10T06:38:38.000Z

I eventually solved the problem. I found that it was the dataset that was causing the error. The correct dataset needs to be downloaded and processed according to README, moving the required images to the appropriate path. After changing the config path I was able to train successfully and get the correct results

I also had this issue, but I couldn't figure out how to solve it. I think it has to do with loading the checkpoint but not sure. I gave up in the end.