Training crashes at 118 epochs
Closed this issue · 5 comments
Hello, when I run the training script and use the training data you provided, the script crashes at 118 epochs with the error message of "Assertion error: (1, index error('too many indices for array: array is 1-dimensional but 2 were indexed') as seen below. Can you please tell me what is the cause of this and how it can be fixed?
Thank you
Does infer.py work well?
Infer.py works well. And now the training code works as well- when I put all the data it works.
In reference to enhancing detection, can an extra class label for walls be added (and annotated in the labelled data)? Does the current architecture allow for this addition?
Of course. But I'm not sure whether window/wall/door detecion can be done in one branch(or network). Maybe you can try multi-branch or other customized architecture.
The same thing happens to me during inference, it seems like function get_coords_from_heatmaps_with_NMS doesn't always find peaks and returns arrays with shape (0,), which causes the error.
It depends on training policy and data quality to learn heatmap peaks. Maybe you can try focal loss and add sigmoid after heatmap outputs.
See https://github.com/xingyizhou/CenterNet/blob/master/src/lib/models/networks/msra_resnet.py
https://github.com/xingyizhou/CenterNet/blob/master/src/lib/models/losses.py#L114