D-ECE Computation

[akhilpm]

Can you provide the code snippet you used for D-ECE computation? I am trying to reproduce the D-ECE results with your checkpoints, but it is not matching. In COCO val dataset, there seems to be many images with no GT boxes. I would like to know how you handled them in D-ECE computation.

[akhtarvision]

Thank you for your interest. Please follow the guidelines here
You need to input det files along with imageids.
COCO val dataset is standard, can you specify what problem you are facing?

[akhilpm]

Thanks for the reply Akhtar. The netcal library needs the confidence and "matched" status for each sample. So, I am not sure where you input det files and imageids. I wrote the code for D-ECE computation as shown below.

There are some images for which no annotation is present (eg image ids: 25593, 41488, 49091, 58636). For now, I simply ignored them in ECE computation. My code to compute D-ECE is as follows:

    def compute_d_ece(val_data_loader, coco_api_val):

	thresh_levels = np.arange(50, 100, 5)/100
	dataset_matched = {thresh:torch.zeros(0).to(device) for thresh in thresh_levels}
	dataset_confidences = torch.zeros(0).to(device)

	for count, (samples, targets) in enumerate(val_data_loader):

	    #get the output for each sample
	    outputs = model(samples)
	    orig_target_sizes = torch.stack([t["orig_size"] for t in targets], dim=0)
	    results = postprocessors['bbox'](outputs, orig_target_sizes)[0]
	    pred_boxes, pred_labels, pred_confidences = results["boxes"], results["labels"], results["scores"]

	    """ target_box_labels and target_boxes are obtained with img ids. Used a batch size of 1 """
	    ann_ids = coco_api_val.getAnnIds(imgIds=targets[0]['image_id'])
	    target = coco_api_val.loadAnns(ann_ids)
	    anno = [obj for obj in target if 'iscrowd' not in obj or obj['iscrowd'] == 0]
	    target_box_labels = torch.tensor([obj["category_id"] for obj in anno], dtype=torch.int64).to(pred_boxes.device)
	    target_boxes = box_ops.box_xywh_to_xyxy([obj["bbox"] for obj in anno])

	    if len(target_box_labels)>0:
		""" used this condition because for some images, there were no GT boxes """
		dataset_confidences = torch.cat([dataset_confidences, pred_confidences], dim=0)
		#compute ious and label match
		ious = torchvision.ops.box_iou(pred_boxes, target_boxes)
		label_match = (pred_labels.unsqueeze(1).repeat(1, target_box_labels.shape[0]) == target_box_labels)

		for thresh in thresh_levels:
		    iou_match = (ious>=thresh)
		    matched = torch.any(label_match*iou_match, dim=1)
		    dataset_matched[thresh] = torch.cat([dataset_matched[thresh], matched], dim=0)

	n_bins = [20]
	ece_thresh_levels = []
	ece = netcal.metrics.ECE(n_bins, detection=True)
	for thresh in thresh_levels:		
	    calibration_score = ece.measure(dataset_confidences.cpu().numpy(), dataset_matched[thresh].cpu().numpy())
	    ece_thresh_levels.append(calibration_score)

	print("Final ECE: [ With NETCAL]: {}".format(np.array(ece_thresh_levels).mean())

This is giving an ECE as follows using your checkpoint of Cal-DETR:
Final ECE: [With NETCAL]: 0.05589909592506961

For MBLS checkpoint, it is around 0.06.
Can you please help me to identify where I am doing wrong in D-ECE computation.

[akhtarvision]

Please go through this code
You can see how it proceed with detections and ids. Also score threshold and number of bins matter to compute D-ECE. Note that our bins setting is 10 and threshold is 0.3 as specified in code. Run for few examples first then scale it for whole test set. Also we have mentioned further settings regarding calibration for detection in the paper and IOU is to be considered as 0.5.

[akhilpm]

Thanks, Akhtar for the clarification. I will work with the example and change the parameters as you mentioned.