IoU/Loss not changing for ICDAR2015 when all labels = "###"

I am trying to train it only for detection. So, I edited the .txt files in the annotations of ICDAR2015 and replaced the last part of each line (the label) in each file with "###".
Previously, it was working fine, but now IoU/loss is NOT changing.
I wonder why is this happening. This code is only for detection, so the labels should not matter. Please help @whai362 !

PS - I don't want labels because I want to train this model on my own dataset which has only bounding boxes, not labels (or equivalent;y, all the labels are equal to "###").

I notice that word ### will result in drawContour in training_masks. Perhaps you can try comment this line:

pan_pp_stable/dataset/ic15.py

Line 364 in d52a93a

cv2.drawContours(training_mask, [bboxes[i]], -1, 0, -1)