Issues Training on Macbook 2019

Question

Issues Training on Macbook 2019

hooping opened this issue 4 years ago · 6 comments

Hello, @sowson thanks for creating this project. It’s a great project!

I hit some issues and want to confirm how to train the YOLO v3 model on MacBook Pro 2019(With below Graphics: AMD Radeon Pro 5300M 4 GB and Intel UHD Graphics 630 1536 MB)

First I tried training without using AMD graphics, just in Makefile, change the GPU=1, without other changes. (It seems in this way, it’s using the Intel Graphics) .Then run “make”, and execute the training job below “./darknet detector train cfg/yolov3.data cfg/yolov3.cfg ./darknet53.conv.74”
The training dataset and parameters has been verified on Nvidia server with this repo(OpenCL for YOLOv3), it works well.

But on MacBook 2019, it hit the issue below:

From iteration 3, the loss became abnormal and then in following iteration, all value became “”NAN . I tried it multiple times, it either has issue in the 2nd iteration or the 3rd iteration.

Details logs as below. Would you advise? Thanks a lot.

Device IDs: 2
Device ID: 0
Device name: Intel(R) UHD Graphics 630
Device vendor: Intel Inc.
Device opencl availability: OpenCL 1.2
Device opencl used: 1.2(Jul 6 2020 11:56:19)
Device double precision: NO
Device max group size: 256
Device address bits: 64
my_data
layer filters size input output
0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BFLOPs
1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BFLOPs
2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BFLOPs
3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BFLOPs
4 res 1 208 x 208 x 64 -> 208 x 208 x 64
5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BFLOPs
6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BFLOPs
7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BFLOPs
8 res 5 104 x 104 x 128 -> 104 x 104 x 128
9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BFLOPs
10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BFLOPs
11 res 8 104 x 104 x 128 -> 104 x 104 x 128
12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BFLOPs
13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs
14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs
15 res 12 52 x 52 x 256 -> 52 x 52 x 256
16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs
17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs
18 res 15 52 x 52 x 256 -> 52 x 52 x 256
19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs
20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs
21 res 18 52 x 52 x 256 -> 52 x 52 x 256
22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs
23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs
24 res 21 52 x 52 x 256 -> 52 x 52 x 256
25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs
26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs
27 res 24 52 x 52 x 256 -> 52 x 52 x 256
28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs
29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs
30 res 27 52 x 52 x 256 -> 52 x 52 x 256
31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs
32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs
33 res 30 52 x 52 x 256 -> 52 x 52 x 256
34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs
35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs
36 res 33 52 x 52 x 256 -> 52 x 52 x 256
37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BFLOPs
38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
40 res 37 26 x 26 x 512 -> 26 x 26 x 512
41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
43 res 40 26 x 26 x 512 -> 26 x 26 x 512
44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
46 res 43 26 x 26 x 512 -> 26 x 26 x 512
47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
49 res 46 26 x 26 x 512 -> 26 x 26 x 512
50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
52 res 49 26 x 26 x 512 -> 26 x 26 x 512
53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
55 res 52 26 x 26 x 512 -> 26 x 26 x 512
56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
58 res 55 26 x 26 x 512 -> 26 x 26 x 512
59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
61 res 58 26 x 26 x 512 -> 26 x 26 x 512
62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BFLOPs
63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs
64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs
65 res 62 13 x 13 x1024 -> 13 x 13 x1024
66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs
67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs
68 res 65 13 x 13 x1024 -> 13 x 13 x1024
69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs
70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs
71 res 68 13 x 13 x1024 -> 13 x 13 x1024
72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs
73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs
74 res 71 13 x 13 x1024 -> 13 x 13 x1024
75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs
76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs
77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs
78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs
79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs
80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs
81 conv 27 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 27 0.009 BFLOPs
82 yolo
83 route 79
84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BFLOPs
85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256
86 route 85 61
87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BFLOPs
88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs
92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs
93 conv 27 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 27 0.019 BFLOPs
94 yolo
95 route 91
96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BFLOPs
97 upsample 4x 26 x 26 x 128 -> 104 x 104 x 128
98 route 97 11
99 conv 128 1 x 1 / 1 104 x 104 x 256 -> 104 x 104 x 128 0.709 BFLOPs
100 conv 256 3 x 3 / 1 104 x 104 x 128 -> 104 x 104 x 256 6.380 BFLOPs
101 conv 128 1 x 1 / 1 104 x 104 x 256 -> 104 x 104 x 128 0.709 BFLOPs
102 conv 256 3 x 3 / 1 104 x 104 x 128 -> 104 x 104 x 256 6.380 BFLOPs
103 conv 128 1 x 1 / 1 104 x 104 x 256 -> 104 x 104 x 128 0.709 BFLOPs
104 conv 256 3 x 3 / 1 104 x 104 x 128 -> 104 x 104 x 256 6.380 BFLOPs
105 conv 27 1 x 1 / 1 104 x 104 x 256 -> 104 x 104 x 27 0.150 BFLOPs
106 yolo
Loading weights from ./darknet53.conv.74...Done!
Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
Saving weights to ./weights//training1.conv.weights
Resizing
576
Loaded: 0.000021 seconds
Region 82 Avg IOU: 0.669766, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.885020, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 1.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.354004, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.366495, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.456132, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.429546, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.598901, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.286972, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.369272, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.544681, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.448270, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.776807, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 1.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.521708, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.512312, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.417380, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.644111, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0
1: 5589.400879, 5589.400879 avg, 0.000000 rate, 120.103433 seconds, 16 images
Loaded: 0.000050 seconds
Region 82 Avg IOU: 0.348210, Class: 0.505208, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.425261, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.412689, Class: 0.505208, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.442617, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.428194, Class: 0.490482, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.460231, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.631038, Class: 0.505208, Obj: 0.493800, No Obj: 0.491735, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.381417, Class: 0.505208, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.500166, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 1.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.625354, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 1.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.645166, Class: 0.490482, Obj: 0.493800, No Obj: 0.491735, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.246722, Class: 0.501013, Obj: 0.490056, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.425910, Class: 0.471722, Obj: 0.492571, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.371427, Class: 0.483262, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.439371, Class: 0.499015, Obj: 0.501104, No Obj: 0.500049, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.513600, Class: 0.483262, Obj: 0.493800, No Obj: 0.491735, .5R: 1.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0
2: 6445.931641, 5675.054199 avg, 0.000000 rate, 120.592475 seconds, 32 images
Loaded: 0.000044 seconds
Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.481045, Class: 0.492256, Obj: 0.472559, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: 1.000000, Obj: 0.500000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 2
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.454574, Class: 0.451919, Obj: 0.437122, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.397417, Class: 0.492256, Obj: 0.472559, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.323605, Class: 0.451919, Obj: 0.437122, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 1.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: 0.352548, Class: 0.451919, Obj: 0.437122, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 0.000000, Obj: 1.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 0.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 0.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: 0.000000, Class: 0.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0
3: 6776393516076498944.000000, 677639324119859200.000000 avg, 0.000000 rate, 120.494957 seconds, 48 images
Loaded: 0.000044 seconds
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
4: nan, nan avg, 0.000000 rate, 118.672016 seconds, 64 images
Loaded: 0.000047 seconds
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1
Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0
Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1

Answer 1 · 2020-09-01T10:24:44.000Z

Hello, just add "-i 1” (i is the index of GPU parameter, add at the end) to your command :D. Thanks! Now you are trying run with lack of memory “Device name: Intel(R) UHD Graphics 630” ;-).

…

On Sep 1, 2020, at 10:52, hooping ***@***.***> wrote: Hello, @sowson <https://github.com/sowson> thanks for creating this project. It’s a great project! I hit some issues and want to confirm how to train the YOLO v3 model on MacBook Pro 2019(With below Graphics: AMD Radeon Pro 5300M 4 GB and Intel UHD Graphics 630 1536 MB) First I tried training without using AMD graphics, just in Makefile, change the GPU=1, without other changes. (It seems in this way, it’s using the Intel Graphics) .Then run “make”, and execute the training job below “./darknet detector train cfg/yolov3.data cfg/yolov3.cfg ./darknet53.conv.74” The training dataset and parameters has been verified on Nvidia server with this repo(OpenCL for YOLOv3), it works well. But on MacBook 2019, it hit the issue below: From iteration 3, the loss became abnormal and then in following iteration, all value became “”NAN . I tried it multiple times, it either has issue in the 2nd iteration or the 3rd iteration. Details logs as below. Would you advise? Thanks a lot. Device IDs: 2 Device ID: 0 Device name: Intel(R) UHD Graphics 630 Device vendor: Intel Inc. Device opencl availability: OpenCL 1.2 Device opencl used: 1.2(Jul 6 2020 11:56:19) Device double precision: NO Device max group size: 256 Device address bits: 64 my_data layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BFLOPs 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BFLOPs 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BFLOPs 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BFLOPs 4 res 1 208 x 208 x 64 -> 208 x 208 x 64 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BFLOPs 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BFLOPs 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BFLOPs 8 res 5 104 x 104 x 128 -> 104 x 104 x 128 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BFLOPs 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BFLOPs 11 res 8 104 x 104 x 128 -> 104 x 104 x 128 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BFLOPs 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 15 res 12 52 x 52 x 256 -> 52 x 52 x 256 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 18 res 15 52 x 52 x 256 -> 52 x 52 x 256 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 21 res 18 52 x 52 x 256 -> 52 x 52 x 256 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 24 res 21 52 x 52 x 256 -> 52 x 52 x 256 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 27 res 24 52 x 52 x 256 -> 52 x 52 x 256 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 30 res 27 52 x 52 x 256 -> 52 x 52 x 256 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 33 res 30 52 x 52 x 256 -> 52 x 52 x 256 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 36 res 33 52 x 52 x 256 -> 52 x 52 x 256 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BFLOPs 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 40 res 37 26 x 26 x 512 -> 26 x 26 x 512 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 43 res 40 26 x 26 x 512 -> 26 x 26 x 512 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 46 res 43 26 x 26 x 512 -> 26 x 26 x 512 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 49 res 46 26 x 26 x 512 -> 26 x 26 x 512 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 52 res 49 26 x 26 x 512 -> 26 x 26 x 512 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 55 res 52 26 x 26 x 512 -> 26 x 26 x 512 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 58 res 55 26 x 26 x 512 -> 26 x 26 x 512 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 61 res 58 26 x 26 x 512 -> 26 x 26 x 512 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BFLOPs 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 65 res 62 13 x 13 x1024 -> 13 x 13 x1024 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 68 res 65 13 x 13 x1024 -> 13 x 13 x1024 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 71 res 68 13 x 13 x1024 -> 13 x 13 x1024 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 74 res 71 13 x 13 x1024 -> 13 x 13 x1024 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 81 conv 27 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 27 0.009 BFLOPs 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BFLOPs 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BFLOPs 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 93 conv 27 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 27 0.019 BFLOPs 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BFLOPs 97 upsample 4x 26 x 26 x 128 -> 104 x 104 x 128 98 route 97 11 99 conv 128 1 x 1 / 1 104 x 104 x 256 -> 104 x 104 x 128 0.709 BFLOPs 100 conv 256 3 x 3 / 1 104 x 104 x 128 -> 104 x 104 x 256 6.380 BFLOPs 101 conv 128 1 x 1 / 1 104 x 104 x 256 -> 104 x 104 x 128 0.709 BFLOPs 102 conv 256 3 x 3 / 1 104 x 104 x 128 -> 104 x 104 x 256 6.380 BFLOPs 103 conv 128 1 x 1 / 1 104 x 104 x 256 -> 104 x 104 x 128 0.709 BFLOPs 104 conv 256 3 x 3 / 1 104 x 104 x 128 -> 104 x 104 x 256 6.380 BFLOPs 105 conv 27 1 x 1 / 1 104 x 104 x 256 -> 104 x 104 x 27 0.150 BFLOPs 106 yolo Loading weights from ./darknet53.conv.74...Done! Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005 Saving weights to ./20200827151209//my_data.start.conv.weights Resizing 576 Loaded: 0.000021 seconds Region 82 Avg IOU: 0.669766, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.885020, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 1.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.354004, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.366495, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.456132, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.429546, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.598901, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.286972, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.369272, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.544681, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.448270, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.776807, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 1.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.521708, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.512312, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.417380, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.644111, Class: 0.500000, Obj: 0.500000, No Obj: 0.500000, .5R: 1.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500000, .5R: nan, .75R: nan, count: 0 1: 5589.400879, 5589.400879 avg, 0.000000 rate, 120.103433 seconds, 16 images Loaded: 0.000050 seconds Region 82 Avg IOU: 0.348210, Class: 0.505208, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.425261, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.412689, Class: 0.505208, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.442617, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.428194, Class: 0.490482, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.460231, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.631038, Class: 0.505208, Obj: 0.493800, No Obj: 0.491735, .5R: 1.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.381417, Class: 0.505208, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.500166, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 1.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.625354, Class: 0.499420, Obj: 0.498816, No Obj: 0.500049, .5R: 1.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.645166, Class: 0.490482, Obj: 0.493800, No Obj: 0.491735, .5R: 1.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.246722, Class: 0.501013, Obj: 0.490056, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.425910, Class: 0.471722, Obj: 0.492571, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.371427, Class: 0.483262, Obj: 0.493800, No Obj: 0.491735, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.491735, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.439371, Class: 0.499015, Obj: 0.501104, No Obj: 0.500049, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519303, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.513600, Class: 0.483262, Obj: 0.493800, No Obj: 0.491735, .5R: 1.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.500049, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.339118, .5R: nan, .75R: nan, count: 0 2: 6445.931641, 5675.054199 avg, 0.000000 rate, 120.592475 seconds, 32 images Loaded: 0.000044 seconds Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.481045, Class: 0.492256, Obj: 0.472559, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: 1.000000, Obj: 0.500000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 2 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.454574, Class: 0.451919, Obj: 0.437122, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.397417, Class: 0.492256, Obj: 0.472559, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.323605, Class: 0.451919, Obj: 0.437122, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 1.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.395062, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: 0.352548, Class: 0.451919, Obj: 0.437122, No Obj: 0.477005, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 0.000000, Obj: 1.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.519444, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 1.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 0.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 0.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: 0.000000, Class: 0.000000, Obj: 0.000000, No Obj: 0.395062, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.477005, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: 0.662166, .5R: nan, .75R: nan, count: 0 3: 6776393516076498944.000000, 677639324119859200.000000 avg, 0.000000 rate, 120.494957 seconds, 48 images Loaded: 0.000044 seconds Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 4: nan, nan avg, 0.000000 rate, 118.672016 seconds, 64 images Loaded: 0.000047 seconds Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 Region 94 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 106 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: nan, .75R: nan, count: 0 Region 82 Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, .5R: 0.000000, .75R: 0.000000, count: 1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIC6NQCRXDGRF6L5BXHCKVDSDSY5VANCNFSM4QROVSUA>.

Answer 2 · 2020-09-02T04:22:24.000Z

@sowson Great! it works now. the training is going smoothly. Thanks a lot.
Btw, do we have the performance comparison between training on AMD and Nvidia? including the training time and the precision/recall/map... I am verifying it by myself now, but just want also check with you on this, thanks!

Answer 3 · 2020-09-02T05:05:59.000Z

@hooping Perf Measurement you can find on my preprint in the very detailed description at https://www.preprints.org/manuscript/202007.0506/v1 If you wanted to check on your own please make following structure of directories.

$HOME/github/sowson/darknet
$HOME/github/sowson/darknet-benchmark
$HOME/github/pjreddie/darknet
$HOME/github/pjreddie/darknet-benchmark

Now clone to all of directory structure, once mine and two times CUDA version.

On the "-benchmark" CUDA version run (in the directory after clone):
git apply ../../sowson/patches/cuda-measure.patch

On the "-benchmark" OpenCL version run (in the directory after clone):
Enable BENCHMARK=1 in Makefile and CMakeLists.txt

Build all the "-benchmark" suffixed solutions.

How does it work? Basically it will run only 1 step on 608x608 size network for CIFAR-10, YOLO2, YOLO3 (examples/classifier.c and examples/detector.c), and CUDA vs OpenCL operations quite detailed.

Enjoy!

Answer 4 · 2020-09-02T11:07:55.000Z

Thanks for sharing! It's very prehensile benchmark. I will also run the measurement in my environment when available.

Answer 5 · 2020-09-15T08:35:38.000Z

@hooping can we close this one?

Answer 6 · 2020-09-22T11:53:06.000Z

Sure, thanks!