Unstable training for lucchi and lucchi_pp datasets
Closed this issue · 4 comments
Dear authors, thank you very much for your excellent paper and the efforts to make the code available. I am trying to reproduce your result on the lucchi and lucchi_pp datasets, finding that jaccard_index is not improving during training (i.e. it reduced quickly to 0.0000e+00 at epoch 0 and stayed unchanged). However, the training process is fine for the kasthuri_pp dataset. And I have tried both SGD and ADAM optimizers, but the problem persisted. The model architecture is the Attention U-net. What could be the problem with this scenario?
The training log is attached as follows:
#####################
TRAIN THE MODEL
#####################
build callbacks done
Epoch 1/360
1/297 [..............................] - ETA: 24:24 - loss: 0.7848 - jaccard_index: 0.1074
2/297 [..............................] - ETA: 26s - loss: 0.6453 - jaccard_index: 0.0728
3/297 [..............................] - ETA: 26s - loss: 1.2558 - jaccard_index: 0.0562
4/297 [..............................] - ETA: 26s - loss: 1.0312 - jaccard_index: 0.0424
5/297 [..............................] - ETA: 26s - loss: 0.9182 - jaccard_index: 0.0357
6/297 [..............................] - ETA: 26s - loss: 0.8367 - jaccard_index: 0.0298
7/297 [..............................] - ETA: 26s - loss: 0.7619 - jaccard_index: 0.0256
8/297 [..............................] - ETA: 26s - loss: 0.6990 - jaccard_index: 0.0224
9/297 [..............................] - ETA: 26s - loss: 0.6426 - jaccard_index: 0.0199
10/297 [>.............................] - ETA: 25s - loss: 0.6153 - jaccard_index: 0.0179
11/297 [>.............................] - ETA: 25s - loss: 0.5751 - jaccard_index: 0.0163
12/297 [>.............................] - ETA: 25s - loss: 0.5460 - jaccard_index: 0.0149
13/297 [>.............................] - ETA: 25s - loss: 0.5177 - jaccard_index: 0.0138
14/297 [>.............................] - ETA: 25s - loss: 0.4983 - jaccard_index: 0.0128
15/297 [>.............................] - ETA: 25s - loss: 0.4713 - jaccard_index: 0.0119
16/297 [>.............................] - ETA: 25s - loss: 0.4528 - jaccard_index: 0.0112
17/297 [>.............................] - ETA: 25s - loss: 0.4358 - jaccard_index: 0.0105
18/297 [>.............................] - ETA: 25s - loss: 0.4341 - jaccard_index: 0.0099
19/297 [>.............................] - ETA: 24s - loss: 0.4269 - jaccard_index: 0.0094
20/297 [=>............................] - ETA: 24s - loss: 0.4184 - jaccard_index: 0.0094
21/297 [=>............................] - ETA: 24s - loss: 0.4093 - jaccard_index: 0.0090
22/297 [=>............................] - ETA: 24s - loss: 0.4009 - jaccard_index: 0.0086
23/297 [=>............................] - ETA: 24s - loss: 0.3968 - jaccard_index: 0.0082
24/297 [=>............................] - ETA: 24s - loss: 0.3923 - jaccard_index: 0.0079
25/297 [=>............................] - ETA: 24s - loss: 0.3873 - jaccard_index: 0.0075
26/297 [=>............................] - ETA: 24s - loss: 0.3802 - jaccard_index: 0.0073
27/297 [=>............................] - ETA: 24s - loss: 0.3717 - jaccard_index: 0.0070
28/297 [=>............................] - ETA: 24s - loss: 0.3627 - jaccard_index: 0.0067
29/297 [=>............................] - ETA: 24s - loss: 0.3624 - jaccard_index: 0.0065
30/297 [==>...........................] - ETA: 24s - loss: 0.3552 - jaccard_index: 0.0063
31/297 [==>...........................] - ETA: 24s - loss: 0.3516 - jaccard_index: 0.0061
32/297 [==>...........................] - ETA: 24s - loss: 0.3504 - jaccard_index: 0.0059
33/297 [==>...........................] - ETA: 24s - loss: 0.3444 - jaccard_index: 0.0057
34/297 [==>...........................] - ETA: 24s - loss: 0.3397 - jaccard_index: 0.0055
35/297 [==>...........................] - ETA: 24s - loss: 0.3401 - jaccard_index: 0.0054
36/297 [==>...........................] - ETA: 24s - loss: 0.3377 - jaccard_index: 0.0052
37/297 [==>...........................] - ETA: 23s - loss: 0.3358 - jaccard_index: 0.0051
38/297 [==>...........................] - ETA: 23s - loss: 0.3375 - jaccard_index: 0.0050
39/297 [==>...........................] - ETA: 23s - loss: 0.3353 - jaccard_index: 0.0048
40/297 [===>..........................] - ETA: 23s - loss: 0.3339 - jaccard_index: 0.0047
41/297 [===>..........................] - ETA: 23s - loss: 0.3311 - jaccard_index: 0.0046
42/297 [===>..........................] - ETA: 23s - loss: 0.3271 - jaccard_index: 0.0045
43/297 [===>..........................] - ETA: 23s - loss: 0.3235 - jaccard_index: 0.0044
44/297 [===>..........................] - ETA: 23s - loss: 0.3217 - jaccard_index: 0.0043
45/297 [===>..........................] - ETA: 23s - loss: 0.3197 - jaccard_index: 0.0042
46/297 [===>..........................] - ETA: 23s - loss: 0.3181 - jaccard_index: 0.0041
47/297 [===>..........................] - ETA: 23s - loss: 0.3162 - jaccard_index: 0.0040
48/297 [===>..........................] - ETA: 22s - loss: 0.3156 - jaccard_index: 0.0039
49/297 [===>..........................] - ETA: 22s - loss: 0.3144 - jaccard_index: 0.0038
50/297 [====>.........................] - ETA: 22s - loss: 0.3118 - jaccard_index: 0.0038
51/297 [====>.........................] - ETA: 22s - loss: 0.3106 - jaccard_index: 0.0037
52/297 [====>.........................] - ETA: 22s - loss: 0.3119 - jaccard_index: 0.0036
53/297 [====>.........................] - ETA: 22s - loss: 0.3093 - jaccard_index: 0.0036
54/297 [====>.........................] - ETA: 22s - loss: 0.3096 - jaccard_index: 0.0035
55/297 [====>.........................] - ETA: 22s - loss: 0.3115 - jaccard_index: 0.0034
56/297 [====>.........................] - ETA: 22s - loss: 0.3104 - jaccard_index: 0.0034
57/297 [====>.........................] - ETA: 22s - loss: 0.3109 - jaccard_index: 0.0033
58/297 [====>.........................] - ETA: 22s - loss: 0.3093 - jaccard_index: 0.0033
59/297 [====>.........................] - ETA: 21s - loss: 0.3086 - jaccard_index: 0.0032
60/297 [=====>........................] - ETA: 21s - loss: 0.3078 - jaccard_index: 0.0031
61/297 [=====>........................] - ETA: 21s - loss: 0.3073 - jaccard_index: 0.0031
62/297 [=====>........................] - ETA: 21s - loss: 0.3077 - jaccard_index: 0.0030
63/297 [=====>........................] - ETA: 21s - loss: 0.3072 - jaccard_index: 0.0030
64/297 [=====>........................] - ETA: 21s - loss: 0.3052 - jaccard_index: 0.0029
65/297 [=====>........................] - ETA: 21s - loss: 0.3047 - jaccard_index: 0.0029
66/297 [=====>........................] - ETA: 21s - loss: 0.3039 - jaccard_index: 0.0029
67/297 [=====>........................] - ETA: 21s - loss: 0.3036 - jaccard_index: 0.0028
68/297 [=====>........................] - ETA: 21s - loss: 0.3026 - jaccard_index: 0.0028
69/297 [=====>........................] - ETA: 20s - loss: 0.3021 - jaccard_index: 0.0027
70/297 [======>.......................] - ETA: 20s - loss: 0.3023 - jaccard_index: 0.0027
71/297 [======>.......................] - ETA: 20s - loss: 0.3023 - jaccard_index: 0.0027
72/297 [======>.......................] - ETA: 20s - loss: 0.3010 - jaccard_index: 0.0026
73/297 [======>.......................] - ETA: 20s - loss: 0.3000 - jaccard_index: 0.0026
74/297 [======>.......................] - ETA: 20s - loss: 0.2989 - jaccard_index: 0.0025
75/297 [======>.......................] - ETA: 20s - loss: 0.2972 - jaccard_index: 0.0025
76/297 [======>.......................] - ETA: 20s - loss: 0.2959 - jaccard_index: 0.0025
77/297 [======>.......................] - ETA: 20s - loss: 0.2938 - jaccard_index: 0.0024
78/297 [======>.......................] - ETA: 20s - loss: 0.2909 - jaccard_index: 0.0024
79/297 [======>.......................] - ETA: 20s - loss: 0.2903 - jaccard_index: 0.0024
80/297 [=======>......................] - ETA: 19s - loss: 0.2880 - jaccard_index: 0.0024
81/297 [=======>......................] - ETA: 19s - loss: 0.2880 - jaccard_index: 0.0023
82/297 [=======>......................] - ETA: 19s - loss: 0.2874 - jaccard_index: 0.0023
83/297 [=======>......................] - ETA: 19s - loss: 0.2860 - jaccard_index: 0.0023
84/297 [=======>......................] - ETA: 19s - loss: 0.2865 - jaccard_index: 0.0022
85/297 [=======>......................] - ETA: 19s - loss: 0.2872 - jaccard_index: 0.0022
86/297 [=======>......................] - ETA: 19s - loss: 0.2869 - jaccard_index: 0.0022
87/297 [=======>......................] - ETA: 19s - loss: 0.2883 - jaccard_index: 0.0022
88/297 [=======>......................] - ETA: 19s - loss: 0.2886 - jaccard_index: 0.0023
89/297 [=======>......................] - ETA: 19s - loss: 0.2887 - jaccard_index: 0.0023
90/297 [========>.....................] - ETA: 18s - loss: 0.2881 - jaccard_index: 0.0023
91/297 [========>.....................] - ETA: 18s - loss: 0.2878 - jaccard_index: 0.0022
92/297 [========>.....................] - ETA: 18s - loss: 0.2870 - jaccard_index: 0.0022
93/297 [========>.....................] - ETA: 18s - loss: 0.2864 - jaccard_index: 0.0022
94/297 [========>.....................] - ETA: 18s - loss: 0.2856 - jaccard_index: 0.0022
95/297 [========>.....................] - ETA: 18s - loss: 0.2857 - jaccard_index: 0.0021
96/297 [========>.....................] - ETA: 18s - loss: 0.2847 - jaccard_index: 0.0021
97/297 [========>.....................] - ETA: 18s - loss: 0.2837 - jaccard_index: 0.0021
98/297 [========>.....................] - ETA: 18s - loss: 0.2829 - jaccard_index: 0.0021
99/297 [=========>....................] - ETA: 18s - loss: 0.2825 - jaccard_index: 0.0021
100/297 [=========>....................] - ETA: 18s - loss: 0.2814 - jaccard_index: 0.0020
101/297 [=========>....................] - ETA: 17s - loss: 0.2810 - jaccard_index: 0.0020
102/297 [=========>....................] - ETA: 17s - loss: 0.2817 - jaccard_index: 0.0020
103/297 [=========>....................] - ETA: 17s - loss: 0.2834 - jaccard_index: 0.0020
104/297 [=========>....................] - ETA: 17s - loss: 0.2831 - jaccard_index: 0.0020
105/297 [=========>....................] - ETA: 17s - loss: 0.2829 - jaccard_index: 0.0019
106/297 [=========>....................] - ETA: 17s - loss: 0.2822 - jaccard_index: 0.0019
107/297 [=========>....................] - ETA: 17s - loss: 0.2824 - jaccard_index: 0.0019
108/297 [=========>....................] - ETA: 17s - loss: 0.2823 - jaccard_index: 0.0019
109/297 [==========>...................] - ETA: 17s - loss: 0.2831 - jaccard_index: 0.0019
110/297 [==========>...................] - ETA: 17s - loss: 0.2824 - jaccard_index: 0.0019
111/297 [==========>...................] - ETA: 17s - loss: 0.2821 - jaccard_index: 0.0019
112/297 [==========>...................] - ETA: 16s - loss: 0.2825 - jaccard_index: 0.0019
113/297 [==========>...................] - ETA: 16s - loss: 0.2823 - jaccard_index: 0.0018
114/297 [==========>...................] - ETA: 16s - loss: 0.2809 - jaccard_index: 0.0018
115/297 [==========>...................] - ETA: 16s - loss: 0.2802 - jaccard_index: 0.0018
116/297 [==========>...................] - ETA: 16s - loss: 0.2801 - jaccard_index: 0.0018
117/297 [==========>...................] - ETA: 16s - loss: 0.2806 - jaccard_index: 0.0018
118/297 [==========>...................] - ETA: 16s - loss: 0.2806 - jaccard_index: 0.0018
119/297 [===========>..................] - ETA: 16s - loss: 0.2801 - jaccard_index: 0.0018
120/297 [===========>..................] - ETA: 16s - loss: 0.2795 - jaccard_index: 0.0017
121/297 [===========>..................] - ETA: 16s - loss: 0.2804 - jaccard_index: 0.0017
122/297 [===========>..................] - ETA: 16s - loss: 0.2795 - jaccard_index: 0.0017
123/297 [===========>..................] - ETA: 15s - loss: 0.2794 - jaccard_index: 0.0017
124/297 [===========>..................] - ETA: 15s - loss: 0.2794 - jaccard_index: 0.0017
125/297 [===========>..................] - ETA: 15s - loss: 0.2793 - jaccard_index: 0.0017
126/297 [===========>..................] - ETA: 15s - loss: 0.2796 - jaccard_index: 0.0017
127/297 [===========>..................] - ETA: 15s - loss: 0.2790 - jaccard_index: 0.0016
128/297 [===========>..................] - ETA: 15s - loss: 0.2794 - jaccard_index: 0.0016
129/297 [============>.................] - ETA: 15s - loss: 0.2794 - jaccard_index: 0.0016
130/297 [============>.................] - ETA: 15s - loss: 0.2792 - jaccard_index: 0.0016
131/297 [============>.................] - ETA: 15s - loss: 0.2792 - jaccard_index: 0.0016
132/297 [============>.................] - ETA: 15s - loss: 0.2782 - jaccard_index: 0.0016
133/297 [============>.................] - ETA: 15s - loss: 0.2779 - jaccard_index: 0.0016
134/297 [============>.................] - ETA: 14s - loss: 0.2778 - jaccard_index: 0.0016
135/297 [============>.................] - ETA: 14s - loss: 0.2782 - jaccard_index: 0.0015
136/297 [============>.................] - ETA: 14s - loss: 0.2780 - jaccard_index: 0.0015
137/297 [============>.................] - ETA: 14s - loss: 0.2775 - jaccard_index: 0.0015
138/297 [============>.................] - ETA: 14s - loss: 0.2771 - jaccard_index: 0.0015
139/297 [=============>................] - ETA: 14s - loss: 0.2769 - jaccard_index: 0.0015
140/297 [=============>................] - ETA: 14s - loss: 0.2764 - jaccard_index: 0.0015
141/297 [=============>................] - ETA: 14s - loss: 0.2766 - jaccard_index: 0.0015
142/297 [=============>................] - ETA: 14s - loss: 0.2773 - jaccard_index: 0.0015
143/297 [=============>................] - ETA: 14s - loss: 0.2773 - jaccard_index: 0.0015
144/297 [=============>................] - ETA: 14s - loss: 0.2765 - jaccard_index: 0.0014
145/297 [=============>................] - ETA: 13s - loss: 0.2760 - jaccard_index: 0.0014
146/297 [=============>................] - ETA: 13s - loss: 0.2760 - jaccard_index: 0.0014
147/297 [=============>................] - ETA: 13s - loss: 0.2756 - jaccard_index: 0.0014
148/297 [=============>................] - ETA: 13s - loss: 0.2749 - jaccard_index: 0.0014
149/297 [==============>...............] - ETA: 13s - loss: 0.2739 - jaccard_index: 0.0014
150/297 [==============>...............] - ETA: 13s - loss: 0.2733 - jaccard_index: 0.0014
151/297 [==============>...............] - ETA: 13s - loss: 0.2748 - jaccard_index: 0.0014
152/297 [==============>...............] - ETA: 13s - loss: 0.2743 - jaccard_index: 0.0014
153/297 [==============>...............] - ETA: 13s - loss: 0.2738 - jaccard_index: 0.0014
154/297 [==============>...............] - ETA: 13s - loss: 0.2737 - jaccard_index: 0.0014
155/297 [==============>...............] - ETA: 12s - loss: 0.2740 - jaccard_index: 0.0013
156/297 [==============>...............] - ETA: 12s - loss: 0.2734 - jaccard_index: 0.0013
157/297 [==============>...............] - ETA: 12s - loss: 0.2733 - jaccard_index: 0.0013
158/297 [==============>...............] - ETA: 12s - loss: 0.2734 - jaccard_index: 0.0013
159/297 [===============>..............] - ETA: 12s - loss: 0.2730 - jaccard_index: 0.0013
160/297 [===============>..............] - ETA: 12s - loss: 0.2727 - jaccard_index: 0.0013
161/297 [===============>..............] - ETA: 12s - loss: 0.2726 - jaccard_index: 0.0013
162/297 [===============>..............] - ETA: 12s - loss: 0.2720 - jaccard_index: 0.0013
163/297 [===============>..............] - ETA: 12s - loss: 0.2721 - jaccard_index: 0.0013
164/297 [===============>..............] - ETA: 12s - loss: 0.2713 - jaccard_index: 0.0013
165/297 [===============>..............] - ETA: 12s - loss: 0.2709 - jaccard_index: 0.0013
166/297 [===============>..............] - ETA: 11s - loss: 0.2702 - jaccard_index: 0.0013
167/297 [===============>..............] - ETA: 11s - loss: 0.2694 - jaccard_index: 0.0012
168/297 [===============>..............] - ETA: 11s - loss: 0.2684 - jaccard_index: 0.0012
169/297 [================>.............] - ETA: 11s - loss: 0.2680 - jaccard_index: 0.0012
170/297 [================>.............] - ETA: 11s - loss: 0.2687 - jaccard_index: 0.0012
171/297 [================>.............] - ETA: 11s - loss: 0.2686 - jaccard_index: 0.0012
172/297 [================>.............] - ETA: 11s - loss: 0.2684 - jaccard_index: 0.0012
173/297 [================>.............] - ETA: 11s - loss: 0.2677 - jaccard_index: 0.0012
174/297 [================>.............] - ETA: 11s - loss: 0.2686 - jaccard_index: 0.0012
175/297 [================>.............] - ETA: 11s - loss: 0.2685 - jaccard_index: 0.0012
176/297 [================>.............] - ETA: 11s - loss: 0.2686 - jaccard_index: 0.0012
177/297 [================>.............] - ETA: 10s - loss: 0.2680 - jaccard_index: 0.0012
178/297 [================>.............] - ETA: 10s - loss: 0.2675 - jaccard_index: 0.0012
179/297 [=================>............] - ETA: 10s - loss: 0.2670 - jaccard_index: 0.0012
180/297 [=================>............] - ETA: 10s - loss: 0.2671 - jaccard_index: 0.0012
181/297 [=================>............] - ETA: 10s - loss: 0.2667 - jaccard_index: 0.0012
182/297 [=================>............] - ETA: 10s - loss: 0.2664 - jaccard_index: 0.0011
183/297 [=================>............] - ETA: 10s - loss: 0.2662 - jaccard_index: 0.0011
184/297 [=================>............] - ETA: 10s - loss: 0.2656 - jaccard_index: 0.0011
185/297 [=================>............] - ETA: 10s - loss: 0.2652 - jaccard_index: 0.0011
186/297 [=================>............] - ETA: 10s - loss: 0.2657 - jaccard_index: 0.0011
187/297 [=================>............] - ETA: 10s - loss: 0.2651 - jaccard_index: 0.0011
188/297 [=================>............] - ETA: 9s - loss: 0.2648 - jaccard_index: 0.0011
189/297 [==================>...........] - ETA: 9s - loss: 0.2649 - jaccard_index: 0.0011
190/297 [==================>...........] - ETA: 9s - loss: 0.2656 - jaccard_index: 0.0011
191/297 [==================>...........] - ETA: 9s - loss: 0.2662 - jaccard_index: 0.0011
192/297 [==================>...........] - ETA: 9s - loss: 0.2661 - jaccard_index: 0.0011
193/297 [==================>...........] - ETA: 9s - loss: 0.2657 - jaccard_index: 0.0011
194/297 [==================>...........] - ETA: 9s - loss: 0.2657 - jaccard_index: 0.0011
195/297 [==================>...........] - ETA: 9s - loss: 0.2656 - jaccard_index: 0.0011
196/297 [==================>...........] - ETA: 9s - loss: 0.2653 - jaccard_index: 0.0011
197/297 [==================>...........] - ETA: 9s - loss: 0.2648 - jaccard_index: 0.0011
198/297 [===================>..........] - ETA: 9s - loss: 0.2647 - jaccard_index: 0.0011
199/297 [===================>..........] - ETA: 8s - loss: 0.2642 - jaccard_index: 0.0010
200/297 [===================>..........] - ETA: 8s - loss: 0.2642 - jaccard_index: 0.0010
201/297 [===================>..........] - ETA: 8s - loss: 0.2643 - jaccard_index: 0.0010
202/297 [===================>..........] - ETA: 8s - loss: 0.2638 - jaccard_index: 0.0010
203/297 [===================>..........] - ETA: 8s - loss: 0.2645 - jaccard_index: 0.0010
204/297 [===================>..........] - ETA: 8s - loss: 0.2650 - jaccard_index: 0.0010
205/297 [===================>..........] - ETA: 8s - loss: 0.2644 - jaccard_index: 0.0010
206/297 [===================>..........] - ETA: 8s - loss: 0.2646 - jaccard_index: 0.0010
207/297 [===================>..........] - ETA: 8s - loss: 0.2644 - jaccard_index: 0.0010
208/297 [====================>.........] - ETA: 8s - loss: 0.2649 - jaccard_index: 0.0010
209/297 [====================>.........] - ETA: 8s - loss: 0.2655 - jaccard_index: 9.9654e-04
210/297 [====================>.........] - ETA: 7s - loss: 0.2652 - jaccard_index: 9.9180e-04
211/297 [====================>.........] - ETA: 7s - loss: 0.2652 - jaccard_index: 9.8710e-04
212/297 [====================>.........] - ETA: 7s - loss: 0.2651 - jaccard_index: 9.8244e-04
213/297 [====================>.........] - ETA: 7s - loss: 0.2650 - jaccard_index: 9.7783e-04
214/297 [====================>.........] - ETA: 7s - loss: 0.2651 - jaccard_index: 9.7326e-04
215/297 [====================>.........] - ETA: 7s - loss: 0.2649 - jaccard_index: 9.6873e-04
216/297 [====================>.........] - ETA: 7s - loss: 0.2644 - jaccard_index: 9.6425e-04
217/297 [====================>.........] - ETA: 7s - loss: 0.2646 - jaccard_index: 9.5980e-04
218/297 [=====================>........] - ETA: 7s - loss: 0.2644 - jaccard_index: 9.5540e-04
219/297 [=====================>........] - ETA: 7s - loss: 0.2644 - jaccard_index: 9.5104e-04
220/297 [=====================>........] - ETA: 7s - loss: 0.2639 - jaccard_index: 9.4671e-04
221/297 [=====================>........] - ETA: 6s - loss: 0.2633 - jaccard_index: 9.4243e-04
222/297 [=====================>........] - ETA: 6s - loss: 0.2634 - jaccard_index: 9.3819e-04
223/297 [=====================>........] - ETA: 6s - loss: 0.2630 - jaccard_index: 9.3398e-04
224/297 [=====================>........] - ETA: 6s - loss: 0.2633 - jaccard_index: 9.2981e-04
225/297 [=====================>........] - ETA: 6s - loss: 0.2630 - jaccard_index: 9.2568e-04
226/297 [=====================>........] - ETA: 6s - loss: 0.2625 - jaccard_index: 9.2158e-04
227/297 [=====================>........] - ETA: 6s - loss: 0.2623 - jaccard_index: 9.1752e-04
228/297 [======================>.......] - ETA: 6s - loss: 0.2623 - jaccard_index: 9.1350e-04
229/297 [======================>.......] - ETA: 6s - loss: 0.2618 - jaccard_index: 9.0951e-04
230/297 [======================>.......] - ETA: 6s - loss: 0.2620 - jaccard_index: 9.0555e-04
231/297 [======================>.......] - ETA: 6s - loss: 0.2615 - jaccard_index: 9.0163e-04
232/297 [======================>.......] - ETA: 5s - loss: 0.2613 - jaccard_index: 8.9775e-04
233/297 [======================>.......] - ETA: 5s - loss: 0.2611 - jaccard_index: 8.9389e-04
234/297 [======================>.......] - ETA: 5s - loss: 0.2605 - jaccard_index: 8.9007e-04
235/297 [======================>.......] - ETA: 5s - loss: 0.2600 - jaccard_index: 8.8629e-04
236/297 [======================>.......] - ETA: 5s - loss: 0.2595 - jaccard_index: 8.8253e-04
237/297 [======================>.......] - ETA: 5s - loss: 0.2593 - jaccard_index: 8.7881e-04
238/297 [=======================>......] - ETA: 5s - loss: 0.2592 - jaccard_index: 8.7511e-04
239/297 [=======================>......] - ETA: 5s - loss: 0.2594 - jaccard_index: 8.7145e-04
240/297 [=======================>......] - ETA: 5s - loss: 0.2592 - jaccard_index: 8.6782e-04
241/297 [=======================>......] - ETA: 5s - loss: 0.2589 - jaccard_index: 8.6422e-04
242/297 [=======================>......] - ETA: 5s - loss: 0.2585 - jaccard_index: 8.6065e-04
243/297 [=======================>......] - ETA: 4s - loss: 0.2580 - jaccard_index: 8.5711e-04
244/297 [=======================>......] - ETA: 4s - loss: 0.2581 - jaccard_index: 8.5359e-04
245/297 [=======================>......] - ETA: 4s - loss: 0.2576 - jaccard_index: 8.5011e-04
246/297 [=======================>......] - ETA: 4s - loss: 0.2580 - jaccard_index: 8.4666e-04
247/297 [=======================>......] - ETA: 4s - loss: 0.2582 - jaccard_index: 8.4323e-04
248/297 [========================>.....] - ETA: 4s - loss: 0.2580 - jaccard_index: 8.3983e-04
249/297 [========================>.....] - ETA: 4s - loss: 0.2581 - jaccard_index: 8.3645e-04
250/297 [========================>.....] - ETA: 4s - loss: 0.2576 - jaccard_index: 8.3311e-04
251/297 [========================>.....] - ETA: 4s - loss: 0.2571 - jaccard_index: 8.2979e-04
252/297 [========================>.....] - ETA: 4s - loss: 0.2574 - jaccard_index: 8.2650e-04
253/297 [========================>.....] - ETA: 4s - loss: 0.2569 - jaccard_index: 8.2323e-04
254/297 [========================>.....] - ETA: 3s - loss: 0.2566 - jaccard_index: 8.1999e-04
255/297 [========================>.....] - ETA: 3s - loss: 0.2568 - jaccard_index: 8.1677e-04
256/297 [========================>.....] - ETA: 3s - loss: 0.2563 - jaccard_index: 8.1358e-04
257/297 [========================>.....] - ETA: 3s - loss: 0.2568 - jaccard_index: 8.1042e-04
258/297 [=========================>....] - ETA: 3s - loss: 0.2566 - jaccard_index: 8.0728e-04
259/297 [=========================>....] - ETA: 3s - loss: 0.2566 - jaccard_index: 8.0416e-04
260/297 [=========================>....] - ETA: 3s - loss: 0.2568 - jaccard_index: 8.0107e-04
261/297 [=========================>....] - ETA: 3s - loss: 0.2569 - jaccard_index: 7.9800e-04
262/297 [=========================>....] - ETA: 3s - loss: 0.2575 - jaccard_index: 7.9495e-04
263/297 [=========================>....] - ETA: 3s - loss: 0.2576 - jaccard_index: 7.9193e-04
264/297 [=========================>....] - ETA: 3s - loss: 0.2575 - jaccard_index: 7.8893e-04
265/297 [=========================>....] - ETA: 2s - loss: 0.2572 - jaccard_index: 7.8595e-04
266/297 [=========================>....] - ETA: 2s - loss: 0.2575 - jaccard_index: 7.8300e-04
267/297 [=========================>....] - ETA: 2s - loss: 0.2573 - jaccard_index: 7.8006e-04
268/297 [==========================>...] - ETA: 2s - loss: 0.2571 - jaccard_index: 7.7715e-04
269/297 [==========================>...] - ETA: 2s - loss: 0.2568 - jaccard_index: 7.7426e-04
270/297 [==========================>...] - ETA: 2s - loss: 0.2568 - jaccard_index: 7.7140e-04
271/297 [==========================>...] - ETA: 2s - loss: 0.2563 - jaccard_index: 7.6855e-04
272/297 [==========================>...] - ETA: 2s - loss: 0.2560 - jaccard_index: 7.6572e-04
273/297 [==========================>...] - ETA: 2s - loss: 0.2556 - jaccard_index: 7.6292e-04
274/297 [==========================>...] - ETA: 2s - loss: 0.2560 - jaccard_index: 7.6014e-04
275/297 [==========================>...] - ETA: 2s - loss: 0.2561 - jaccard_index: 7.5737e-04
276/297 [==========================>...] - ETA: 1s - loss: 0.2559 - jaccard_index: 7.5463e-04
277/297 [==========================>...] - ETA: 1s - loss: 0.2558 - jaccard_index: 7.5190e-04
278/297 [===========================>..] - ETA: 1s - loss: 0.2556 - jaccard_index: 7.4920e-04
279/297 [===========================>..] - ETA: 1s - loss: 0.2557 - jaccard_index: 7.4651e-04
280/297 [===========================>..] - ETA: 1s - loss: 0.2558 - jaccard_index: 7.4385e-04
281/297 [===========================>..] - ETA: 1s - loss: 0.2559 - jaccard_index: 7.4271e-04
282/297 [===========================>..] - ETA: 1s - loss: 0.2556 - jaccard_index: 7.4008e-04
283/297 [===========================>..] - ETA: 1s - loss: 0.2552 - jaccard_index: 7.3746e-04
284/297 [===========================>..] - ETA: 1s - loss: 0.2546 - jaccard_index: 7.3487e-04
285/297 [===========================>..] - ETA: 1s - loss: 0.2550 - jaccard_index: 7.3229e-04
286/297 [===========================>..] - ETA: 1s - loss: 0.2546 - jaccard_index: 7.2973e-04
287/297 [===========================>..] - ETA: 0s - loss: 0.2546 - jaccard_index: 7.2718e-04
288/297 [============================>.] - ETA: 0s - loss: 0.2547 - jaccard_index: 7.2466e-04
289/297 [============================>.] - ETA: 0s - loss: 0.2544 - jaccard_index: 7.2215e-04
290/297 [============================>.] - ETA: 0s - loss: 0.2541 - jaccard_index: 7.1966e-04
291/297 [============================>.] - ETA: 0s - loss: 0.2545 - jaccard_index: 7.1719e-04
292/297 [============================>.] - ETA: 0s - loss: 0.2544 - jaccard_index: 7.1473e-04
293/297 [============================>.] - ETA: 0s - loss: 0.2545 - jaccard_index: 7.1229e-04
294/297 [============================>.] - ETA: 0s - loss: 0.2549 - jaccard_index: 7.0987e-04
295/297 [============================>.] - ETA: 0s - loss: 0.2547 - jaccard_index: 7.0746e-04
296/297 [============================>.] - ETA: 0s - loss: 0.2548 - jaccard_index: 7.0507e-04
297/297 [==============================] - ETA: 0s - loss: 0.2547 - jaccard_index: 7.0270e-04
Epoch 00001: val_loss improved from inf to 0.25942, saving model to content\output\reproduce_lucchi_pp_10\checkpoints\model_weights_reproduce_lucchi_pp_10_10.h5
297/297 [==============================] - 34s 98ms/step - loss: 0.2547 - jaccard_index: 7.0270e-04 - val_loss: 0.2594 - val_jaccard_index: 0.0000e+00
Epoch 2/360
1/297 [..............................] - ETA: 39s - loss: 0.3499 - jaccard_index: 0.0000e+00
2/297 [..............................] - ETA: 26s - loss: 0.2899 - jaccard_index: 0.0000e+00
3/297 [..............................] - ETA: 26s - loss: 0.2773 - jaccard_index: 0.0000e+00
4/297 [..............................] - ETA: 26s - loss: 0.2892 - jaccard_index: 0.0000e+00
5/297 [..............................] - ETA: 26s - loss: 0.2869 - jaccard_index: 0.0000e+00
6/297 [..............................] - ETA: 25s - loss: 0.3050 - jaccard_index: 0.0000e+00
7/297 [..............................] - ETA: 25s - loss: 0.2988 - jaccard_index: 0.0000e+00
8/297 [..............................] - ETA: 25s - loss: 0.2908 - jaccard_index: 0.0000e+00
9/297 [..............................] - ETA: 26s - loss: 0.2805 - jaccard_index: 0.0000e+00
10/297 [>.............................] - ETA: 26s - loss: 0.2854 - jaccard_index: 0.0000e+00
11/297 [>.............................] - ETA: 25s - loss: 0.2756 - jaccard_index: 0.0000e+00
12/297 [>.............................] - ETA: 25s - loss: 0.2721 - jaccard_index: 0.0000e+00
13/297 [>.............................] - ETA: 25s - loss: 0.2647 - jaccard_index: 0.0000e+00
14/297 [>.............................] - ETA: 26s - loss: 0.2620 - jaccard_index: 0.0000e+00
15/297 [>.............................] - ETA: 26s - loss: 0.2517 - jaccard_index: 0.0000e+00
16/297 [>.............................] - ETA: 26s - loss: 0.2469 - jaccard_index: 0.0000e+00
17/297 [>.............................] - ETA: 26s - loss: 0.2415 - jaccard_index: 0.0000e+00
18/297 [>.............................] - ETA: 25s - loss: 0.2484 - jaccard_index: 0.0000e+00
19/297 [>.............................] - ETA: 25s - loss: 0.2506 - jaccard_index: 0.0000e+00
20/297 [=>............................] - ETA: 25s - loss: 0.2493 - jaccard_index: 0.0000e+00
21/297 [=>............................] - ETA: 25s - loss: 0.2488 - jaccard_index: 0.0000e+00
22/297 [=>............................] - ETA: 25s - loss: 0.2465 - jaccard_index: 0.0000e+00
23/297 [=>............................] - ETA: 25s - loss: 0.2478 - jaccard_index: 0.0000e+00
24/297 [=>............................] - ETA: 25s - loss: 0.2494 - jaccard_index: 0.0000e+00
25/297 [=>............................] - ETA: 25s - loss: 0.2502 - jaccard_index: 0.0000e+00
26/297 [=>............................] - ETA: 25s - loss: 0.2479 - jaccard_index: 0.0000e+00
27/297 [=>............................] - ETA: 25s - loss: 0.2434 - jaccard_index: 0.0000e+00
28/297 [=>............................] - ETA: 25s - loss: 0.2394 - jaccard_index: 0.0000e+00
29/297 [=>............................] - ETA: 25s - loss: 0.2439 - jaccard_index: 0.0000e+00
30/297 [==>...........................] - ETA: 24s - loss: 0.2408 - jaccard_index: 0.0000e+00
31/297 [==>...........................] - ETA: 24s - loss: 0.2397 - jaccard_index: 0.0000e+00
32/297 [==>...........................] - ETA: 24s - loss: 0.2411 - jaccard_index: 0.0000e+00
33/297 [==>...........................] - ETA: 24s - loss: 0.2387 - jaccard_index: 0.0000e+00
34/297 [==>...........................] - ETA: 24s - loss: 0.2369 - jaccard_index: 0.0000e+00
35/297 [==>...........................] - ETA: 24s - loss: 0.2398 - jaccard_index: 0.0000e+00
36/297 [==>...........................] - ETA: 24s - loss: 0.2397 - jaccard_index: 0.0000e+00
37/297 [==>...........................] - ETA: 24s - loss: 0.2408 - jaccard_index: 0.0000e+00
38/297 [==>...........................] - ETA: 24s - loss: 0.2446 - jaccard_index: 0.0000e+00
39/297 [==>...........................] - ETA: 23s - loss: 0.2446 - jaccard_index: 0.0000e+00
40/297 [===>..........................] - ETA: 23s - loss: 0.2452 - jaccard_index: 0.0000e+00
41/297 [===>..........................] - ETA: 23s - loss: 0.2443 - jaccard_index: 0.0000e+00
42/297 [===>..........................] - ETA: 23s - loss: 0.2417 - jaccard_index: 0.0000e+00
43/297 [===>..........................] - ETA: 23s - loss: 0.2399 - jaccard_index: 0.0000e+00
44/297 [===>..........................] - ETA: 23s - loss: 0.2399 - jaccard_index: 0.0000e+00
45/297 [===>..........................] - ETA: 23s - loss: 0.2394 - jaccard_index: 0.0000e+00
46/297 [===>..........................] - ETA: 23s - loss: 0.2394 - jaccard_index: 0.0000e+00
47/297 [===>..........................] - ETA: 23s - loss: 0.2389 - jaccard_index: 0.0000e+00
48/297 [===>..........................] - ETA: 23s - loss: 0.2398 - jaccard_index: 0.0000e+00
49/297 [===>..........................] - ETA: 22s - loss: 0.2398 - jaccard_index: 0.0000e+00
50/297 [====>.........................] - ETA: 22s - loss: 0.2387 - jaccard_index: 0.0000e+00
51/297 [====>.........................] - ETA: 22s - loss: 0.2390 - jaccard_index: 0.0000e+00
52/297 [====>.........................] - ETA: 22s - loss: 0.2411 - jaccard_index: 0.0000e+00
53/297 [====>.........................] - ETA: 22s - loss: 0.2398 - jaccard_index: 0.0000e+00
54/297 [====>.........................] - ETA: 22s - loss: 0.2410 - jaccard_index: 0.0000e+00
55/297 [====>.........................] - ETA: 22s - loss: 0.2440 - jaccard_index: 0.0000e+00
56/297 [====>.........................] - ETA: 22s - loss: 0.2441 - jaccard_index: 0.0000e+00
57/297 [====>.........................] - ETA: 22s - loss: 0.2454 - jaccard_index: 0.0000e+00
58/297 [====>.........................] - ETA: 22s - loss: 0.2447 - jaccard_index: 0.0000e+00
59/297 [====>.........................] - ETA: 22s - loss: 0.2455 - jaccard_index: 0.0000e+00
60/297 [=====>........................] - ETA: 21s - loss: 0.2459 - jaccard_index: 0.0000e+00
61/297 [=====>........................] - ETA: 21s - loss: 0.2464 - jaccard_index: 0.0000e+00
62/297 [=====>........................] - ETA: 21s - loss: 0.2477 - jaccard_index: 0.0000e+00
63/297 [=====>........................] - ETA: 21s - loss: 0.2481 - jaccard_index: 0.0000e+00
64/297 [=====>........................] - ETA: 21s - loss: 0.2472 - jaccard_index: 0.0000e+00
65/297 [=====>........................] - ETA: 21s - loss: 0.2475 - jaccard_index: 0.0000e+00
66/297 [=====>........................] - ETA: 21s - loss: 0.2475 - jaccard_index: 0.0000e+00
67/297 [=====>........................] - ETA: 21s - loss: 0.2477 - jaccard_index: 0.0000e+00
68/297 [=====>........................] - ETA: 21s - loss: 0.2477 - jaccard_index: 0.0000e+00
69/297 [=====>........................] - ETA: 21s - loss: 0.2480 - jaccard_index: 0.0000e+00
70/297 [======>.......................] - ETA: 20s - loss: 0.2488 - jaccard_index: 0.0000e+00
71/297 [======>.......................] - ETA: 20s - loss: 0.2493 - jaccard_index: 0.0000e+00
72/297 [======>.......................] - ETA: 20s - loss: 0.2485 - jaccard_index: 0.0000e+00
73/297 [======>.......................] - ETA: 20s - loss: 0.2481 - jaccard_index: 0.0000e+00
74/297 [======>.......................] - ETA: 20s - loss: 0.2474 - jaccard_index: 0.0000e+00
75/297 [======>.......................] - ETA: 20s - loss: 0.2465 - jaccard_index: 0.0000e+00
76/297 [======>.......................] - ETA: 20s - loss: 0.2459 - jaccard_index: 0.0000e+00
77/297 [======>.......................] - ETA: 20s - loss: 0.2444 - jaccard_index: 0.0000e+00
78/297 [======>.......................] - ETA: 20s - loss: 0.2421 - jaccard_index: 0.0000e+00
79/297 [======>.......................] - ETA: 20s - loss: 0.2420 - jaccard_index: 0.0000e+00
80/297 [=======>......................] - ETA: 19s - loss: 0.2403 - jaccard_index: 0.0000e+00
81/297 [=======>......................] - ETA: 19s - loss: 0.2408 - jaccard_index: 0.0000e+00
82/297 [=======>......................] - ETA: 19s - loss: 0.2409 - jaccard_index: 0.0000e+00
83/297 [=======>......................] - ETA: 19s - loss: 0.2401 - jaccard_index: 0.0000e+00
84/297 [=======>......................] - ETA: 19s - loss: 0.2411 - jaccard_index: 0.0000e+00
85/297 [=======>......................] - ETA: 19s - loss: 0.2421 - jaccard_index: 0.0000e+00
86/297 [=======>......................] - ETA: 19s - loss: 0.2427 - jaccard_index: 0.0000e+00
87/297 [=======>......................] - ETA: 19s - loss: 0.2449 - jaccard_index: 0.0000e+00
88/297 [=======>......................] - ETA: 19s - loss: 0.2453 - jaccard_index: 0.0000e+00
89/297 [=======>......................] - ETA: 19s - loss: 0.2456 - jaccard_index: 0.0000e+00
90/297 [========>.....................] - ETA: 19s - loss: 0.2453 - jaccard_index: 0.0000e+00
91/297 [========>.....................] - ETA: 18s - loss: 0.2455 - jaccard_index: 0.0000e+00
92/297 [========>.....................] - ETA: 18s - loss: 0.2452 - jaccard_index: 0.0000e+00
93/297 [========>.....................] - ETA: 18s - loss: 0.2452 - jaccard_index: 0.0000e+00
94/297 [========>.....................] - ETA: 18s - loss: 0.2447 - jaccard_index: 0.0000e+00
95/297 [========>.....................] - ETA: 18s - loss: 0.2453 - jaccard_index: 0.0000e+00
96/297 [========>.....................] - ETA: 18s - loss: 0.2448 - jaccard_index: 0.0000e+00
97/297 [========>.....................] - ETA: 18s - loss: 0.2442 - jaccard_index: 0.0000e+00
98/297 [========>.....................] - ETA: 18s - loss: 0.2435 - jaccard_index: 0.0000e+00
99/297 [=========>....................] - ETA: 18s - loss: 0.2436 - jaccard_index: 0.0000e+00
100/297 [=========>....................] - ETA: 18s - loss: 0.2430 - jaccard_index: 0.0000e+00
101/297 [=========>....................] - ETA: 18s - loss: 0.2427 - jaccard_index: 0.0000e+00
102/297 [=========>....................] - ETA: 17s - loss: 0.2437 - jaccard_index: 0.0000e+00
103/297 [=========>....................] - ETA: 17s - loss: 0.2458 - jaccard_index: 0.0000e+00
104/297 [=========>....................] - ETA: 17s - loss: 0.2458 - jaccard_index: 0.0000e+00
105/297 [=========>....................] - ETA: 17s - loss: 0.2459 - jaccard_index: 0.0000e+00
106/297 [=========>....................] - ETA: 17s - loss: 0.2456 - jaccard_index: 0.0000e+00
107/297 [=========>....................] - ETA: 17s - loss: 0.2461 - jaccard_index: 0.0000e+00
108/297 [=========>....................] - ETA: 17s - loss: 0.2464 - jaccard_index: 0.0000e+00
109/297 [==========>...................] - ETA: 17s - loss: 0.2474 - jaccard_index: 0.0000e+00
110/297 [==========>...................] - ETA: 17s - loss: 0.2467 - jaccard_index: 0.0000e+00
111/297 [==========>...................] - ETA: 17s - loss: 0.2467 - jaccard_index: 0.0000e+00
112/297 [==========>...................] - ETA: 16s - loss: 0.2474 - jaccard_index: 0.0000e+00
113/297 [==========>...................] - ETA: 16s - loss: 0.2474 - jaccard_index: 0.0000e+00
114/297 [==========>...................] - ETA: 16s - loss: 0.2464 - jaccard_index: 0.0000e+00
115/297 [==========>...................] - ETA: 16s - loss: 0.2459 - jaccard_index: 0.0000e+00
116/297 [==========>...................] - ETA: 16s - loss: 0.2460 - jaccard_index: 0.0000e+00
117/297 [==========>...................] - ETA: 16s - loss: 0.2466 - jaccard_index: 0.0000e+00
118/297 [==========>...................] - ETA: 16s - loss: 0.2469 - jaccard_index: 0.0000e+00
119/297 [===========>..................] - ETA: 16s - loss: 0.2467 - jaccard_index: 0.0000e+00
120/297 [===========>..................] - ETA: 16s - loss: 0.2465 - jaccard_index: 0.0000e+00
121/297 [===========>..................] - ETA: 16s - loss: 0.2478 - jaccard_index: 0.0000e+00
122/297 [===========>..................] - ETA: 16s - loss: 0.2471 - jaccard_index: 0.0000e+00
123/297 [===========>..................] - ETA: 15s - loss: 0.2473 - jaccard_index: 0.0000e+00
124/297 [===========>..................] - ETA: 15s - loss: 0.2476 - jaccard_index: 0.0000e+00
125/297 [===========>..................] - ETA: 15s - loss: 0.2477 - jaccard_index: 0.0000e+00
126/297 [===========>..................] - ETA: 15s - loss: 0.2481 - jaccard_index: 0.0000e+00
127/297 [===========>..................] - ETA: 15s - loss: 0.2477 - jaccard_index: 0.0000e+00
128/297 [===========>..................] - ETA: 15s - loss: 0.2482 - jaccard_index: 0.0000e+00
129/297 [============>.................] - ETA: 15s - loss: 0.2485 - jaccard_index: 0.0000e+00
130/297 [============>.................] - ETA: 15s - loss: 0.2486 - jaccard_index: 0.0000e+00
131/297 [============>.................] - ETA: 15s - loss: 0.2489 - jaccard_index: 0.0000e+00
132/297 [============>.................] - ETA: 15s - loss: 0.2482 - jaccard_index: 0.0000e+00
133/297 [============>.................] - ETA: 15s - loss: 0.2480 - jaccard_index: 0.0000e+00
134/297 [============>.................] - ETA: 14s - loss: 0.2482 - jaccard_index: 0.0000e+00
135/297 [============>.................] - ETA: 14s - loss: 0.2488 - jaccard_index: 0.0000e+00
136/297 [============>.................] - ETA: 14s - loss: 0.2488 - jaccard_index: 0.0000e+00
137/297 [============>.................] - ETA: 14s - loss: 0.2486 - jaccard_index: 0.0000e+00
138/297 [============>.................] - ETA: 14s - loss: 0.2484 - jaccard_index: 0.0000e+00
139/297 [=============>................] - ETA: 14s - loss: 0.2482 - jaccard_index: 0.0000e+00
140/297 [=============>................] - ETA: 14s - loss: 0.2479 - jaccard_index: 0.0000e+00
141/297 [=============>................] - ETA: 14s - loss: 0.2483 - jaccard_index: 0.0000e+00
142/297 [=============>................] - ETA: 14s - loss: 0.2493 - jaccard_index: 0.0000e+00
143/297 [=============>................] - ETA: 14s - loss: 0.2495 - jaccard_index: 0.0000e+00
144/297 [=============>................] - ETA: 14s - loss: 0.2489 - jaccard_index: 0.0000e+00
145/297 [=============>................] - ETA: 13s - loss: 0.2484 - jaccard_index: 0.0000e+00
146/297 [=============>................] - ETA: 13s - loss: 0.2487 - jaccard_index: 0.0000e+00
147/297 [=============>................] - ETA: 13s - loss: 0.2484 - jaccard_index: 0.0000e+00
148/297 [=============>................] - ETA: 13s - loss: 0.2478 - jaccard_index: 0.0000e+00
149/297 [==============>...............] - ETA: 13s - loss: 0.2469 - jaccard_index: 0.0000e+00
150/297 [==============>...............] - ETA: 13s - loss: 0.2464 - jaccard_index: 0.0000e+00
151/297 [==============>...............] - ETA: 13s - loss: 0.2480 - jaccard_index: 0.0000e+00
152/297 [==============>...............] - ETA: 13s - loss: 0.2475 - jaccard_index: 0.0000e+00
153/297 [==============>...............] - ETA: 13s - loss: 0.2473 - jaccard_index: 0.0000e+00
154/297 [==============>...............] - ETA: 13s - loss: 0.2473 - jaccard_index: 0.0000e+00
155/297 [==============>...............] - ETA: 13s - loss: 0.2477 - jaccard_index: 0.0000e+00
156/297 [==============>...............] - ETA: 12s - loss: 0.2473 - jaccard_index: 0.0000e+00
157/297 [==============>...............] - ETA: 12s - loss: 0.2474 - jaccard_index: 0.0000e+00
158/297 [==============>...............] - ETA: 12s - loss: 0.2476 - jaccard_index: 0.0000e+00
159/297 [===============>..............] - ETA: 12s - loss: 0.2474 - jaccard_index: 0.0000e+00
160/297 [===============>..............] - ETA: 12s - loss: 0.2471 - jaccard_index: 0.0000e+00
161/297 [===============>..............] - ETA: 12s - loss: 0.2472 - jaccard_index: 0.0000e+00
162/297 [===============>..............] - ETA: 12s - loss: 0.2466 - jaccard_index: 0.0000e+00
163/297 [===============>..............] - ETA: 12s - loss: 0.2469 - jaccard_index: 0.0000e+00
164/297 [===============>..............] - ETA: 12s - loss: 0.2461 - jaccard_index: 0.0000e+00
165/297 [===============>..............] - ETA: 12s - loss: 0.2459 - jaccard_index: 0.0000e+00
166/297 [===============>..............] - ETA: 12s - loss: 0.2454 - jaccard_index: 0.0000e+00
167/297 [===============>..............] - ETA: 11s - loss: 0.2446 - jaccard_index: 0.0000e+00
168/297 [===============>..............] - ETA: 11s - loss: 0.2438 - jaccard_index: 0.0000e+00
169/297 [================>.............] - ETA: 11s - loss: 0.2435 - jaccard_index: 0.0000e+00
170/297 [================>.............] - ETA: 11s - loss: 0.2443 - jaccard_index: 0.0000e+00
171/297 [================>.............] - ETA: 11s - loss: 0.2442 - jaccard_index: 0.0000e+00
172/297 [================>.............] - ETA: 11s - loss: 0.2441 - jaccard_index: 0.0000e+00
173/297 [================>.............] - ETA: 11s - loss: 0.2437 - jaccard_index: 0.0000e+00
174/297 [================>.............] - ETA: 11s - loss: 0.2446 - jaccard_index: 0.0000e+00
175/297 [================>.............] - ETA: 11s - loss: 0.2448 - jaccard_index: 0.0000e+00
176/297 [================>.............] - ETA: 11s - loss: 0.2451 - jaccard_index: 0.0000e+00
177/297 [================>.............] - ETA: 11s - loss: 0.2447 - jaccard_index: 0.0000e+00
178/297 [================>.............] - ETA: 10s - loss: 0.2444 - jaccard_index: 0.0000e+00
179/297 [=================>............] - ETA: 10s - loss: 0.2439 - jaccard_index: 0.0000e+00
180/297 [=================>............] - ETA: 10s - loss: 0.2440 - jaccard_index: 0.0000e+00
181/297 [=================>............] - ETA: 10s - loss: 0.2436 - jaccard_index: 0.0000e+00
182/297 [=================>............] - ETA: 10s - loss: 0.2434 - jaccard_index: 0.0000e+00
183/297 [=================>............] - ETA: 10s - loss: 0.2432 - jaccard_index: 0.0000e+00
184/297 [=================>............] - ETA: 10s - loss: 0.2427 - jaccard_index: 0.0000e+00
185/297 [=================>............] - ETA: 10s - loss: 0.2425 - jaccard_index: 0.0000e+00
186/297 [=================>............] - ETA: 10s - loss: 0.2431 - jaccard_index: 0.0000e+00
187/297 [=================>............] - ETA: 10s - loss: 0.2426 - jaccard_index: 0.0000e+00
188/297 [=================>............] - ETA: 10s - loss: 0.2423 - jaccard_index: 0.0000e+00
189/297 [==================>...........] - ETA: 9s - loss: 0.2425 - jaccard_index: 0.0000e+00
190/297 [==================>...........] - ETA: 9s - loss: 0.2431 - jaccard_index: 0.0000e+00
191/297 [==================>...........] - ETA: 9s - loss: 0.2437 - jaccard_index: 0.0000e+00
192/297 [==================>...........] - ETA: 9s - loss: 0.2438 - jaccard_index: 0.0000e+00
193/297 [==================>...........] - ETA: 9s - loss: 0.2434 - jaccard_index: 0.0000e+00
194/297 [==================>...........] - ETA: 9s - loss: 0.2436 - jaccard_index: 0.0000e+00
195/297 [==================>...........] - ETA: 9s - loss: 0.2436 - jaccard_index: 0.0000e+00
196/297 [==================>...........] - ETA: 9s - loss: 0.2434 - jaccard_index: 0.0000e+00
197/297 [==================>...........] - ETA: 9s - loss: 0.2430 - jaccard_index: 0.0000e+00
198/297 [===================>..........] - ETA: 9s - loss: 0.2430 - jaccard_index: 0.0000e+00
199/297 [===================>..........] - ETA: 8s - loss: 0.2424 - jaccard_index: 0.0000e+00
200/297 [===================>..........] - ETA: 8s - loss: 0.2425 - jaccard_index: 0.0000e+00
201/297 [===================>..........] - ETA: 8s - loss: 0.2427 - jaccard_index: 0.0000e+00
202/297 [===================>..........] - ETA: 8s - loss: 0.2423 - jaccard_index: 0.0000e+00
203/297 [===================>..........] - ETA: 8s - loss: 0.2431 - jaccard_index: 0.0000e+00
204/297 [===================>..........] - ETA: 8s - loss: 0.2436 - jaccard_index: 0.0000e+00
205/297 [===================>..........] - ETA: 8s - loss: 0.2433 - jaccard_index: 0.0000e+00
206/297 [===================>..........] - ETA: 8s - loss: 0.2436 - jaccard_index: 0.0000e+00
207/297 [===================>..........] - ETA: 8s - loss: 0.2436 - jaccard_index: 0.0000e+00
208/297 [====================>.........] - ETA: 8s - loss: 0.2441 - jaccard_index: 0.0000e+00
209/297 [====================>.........] - ETA: 8s - loss: 0.2448 - jaccard_index: 0.0000e+00
210/297 [====================>.........] - ETA: 7s - loss: 0.2447 - jaccard_index: 0.0000e+00
211/297 [====================>.........] - ETA: 7s - loss: 0.2449 - jaccard_index: 0.0000e+00
212/297 [====================>.........] - ETA: 7s - loss: 0.2449 - jaccard_index: 0.0000e+00
213/297 [====================>.........] - ETA: 7s - loss: 0.2449 - jaccard_index: 0.0000e+00
214/297 [====================>.........] - ETA: 7s - loss: 0.2451 - jaccard_index: 0.0000e+00
215/297 [====================>.........] - ETA: 7s - loss: 0.2450 - jaccard_index: 0.0000e+00
216/297 [====================>.........] - ETA: 7s - loss: 0.2444 - jaccard_index: 0.0000e+00
217/297 [====================>.........] - ETA: 7s - loss: 0.2448 - jaccard_index: 0.0000e+00
218/297 [=====================>........] - ETA: 7s - loss: 0.2447 - jaccard_index: 0.0000e+00
219/297 [=====================>........] - ETA: 7s - loss: 0.2448 - jaccard_index: 0.0000e+00
220/297 [=====================>........] - ETA: 7s - loss: 0.2445 - jaccard_index: 0.0000e+00
221/297 [=====================>........] - ETA: 6s - loss: 0.2439 - jaccard_index: 0.0000e+00
222/297 [=====================>........] - ETA: 6s - loss: 0.2442 - jaccard_index: 0.0000e+00
223/297 [=====================>........] - ETA: 6s - loss: 0.2437 - jaccard_index: 0.0000e+00
224/297 [=====================>........] - ETA: 6s - loss: 0.2441 - jaccard_index: 0.0000e+00
225/297 [=====================>........] - ETA: 6s - loss: 0.2438 - jaccard_index: 0.0000e+00
226/297 [=====================>........] - ETA: 6s - loss: 0.2435 - jaccard_index: 0.0000e+00
227/297 [=====================>........] - ETA: 6s - loss: 0.2434 - jaccard_index: 0.0000e+00
228/297 [======================>.......] - ETA: 6s - loss: 0.2434 - jaccard_index: 0.0000e+00
229/297 [======================>.......] - ETA: 6s - loss: 0.2430 - jaccard_index: 0.0000e+00
230/297 [======================>.......] - ETA: 6s - loss: 0.2432 - jaccard_index: 0.0000e+00
231/297 [======================>.......] - ETA: 6s - loss: 0.2429 - jaccard_index: 0.0000e+00
232/297 [======================>.......] - ETA: 5s - loss: 0.2428 - jaccard_index: 0.0000e+00
233/297 [======================>.......] - ETA: 5s - loss: 0.2427 - jaccard_index: 0.0000e+00
234/297 [======================>.......] - ETA: 5s - loss: 0.2421 - jaccard_index: 0.0000e+00
235/297 [======================>.......] - ETA: 5s - loss: 0.2416 - jaccard_index: 0.0000e+00
236/297 [======================>.......] - ETA: 5s - loss: 0.2413 - jaccard_index: 0.0000e+00
237/297 [======================>.......] - ETA: 5s - loss: 0.2410 - jaccard_index: 0.0000e+00
238/297 [=======================>......] - ETA: 5s - loss: 0.2410 - jaccard_index: 0.0000e+00
239/297 [=======================>......] - ETA: 5s - loss: 0.2414 - jaccard_index: 0.0000e+00
240/297 [=======================>......] - ETA: 5s - loss: 0.2412 - jaccard_index: 0.0000e+00
241/297 [=======================>......] - ETA: 5s - loss: 0.2410 - jaccard_index: 0.0000e+00
242/297 [=======================>......] - ETA: 5s - loss: 0.2407 - jaccard_index: 0.0000e+00
243/297 [=======================>......] - ETA: 4s - loss: 0.2403 - jaccard_index: 0.0000e+00
244/297 [=======================>......] - ETA: 4s - loss: 0.2405 - jaccard_index: 0.0000e+00
245/297 [=======================>......] - ETA: 4s - loss: 0.2401 - jaccard_index: 0.0000e+00
246/297 [=======================>......] - ETA: 4s - loss: 0.2405 - jaccard_index: 0.0000e+00
247/297 [=======================>......] - ETA: 4s - loss: 0.2406 - jaccard_index: 0.0000e+00
248/297 [========================>.....] - ETA: 4s - loss: 0.2406 - jaccard_index: 0.0000e+00
249/297 [========================>.....] - ETA: 4s - loss: 0.2408 - jaccard_index: 0.0000e+00
250/297 [========================>.....] - ETA: 4s - loss: 0.2403 - jaccard_index: 0.0000e+00
251/297 [========================>.....] - ETA: 4s - loss: 0.2398 - jaccard_index: 0.0000e+00
252/297 [========================>.....] - ETA: 4s - loss: 0.2403 - jaccard_index: 0.0000e+00
253/297 [========================>.....] - ETA: 4s - loss: 0.2397 - jaccard_index: 0.0000e+00
254/297 [========================>.....] - ETA: 3s - loss: 0.2395 - jaccard_index: 0.0000e+00
255/297 [========================>.....] - ETA: 3s - loss: 0.2397 - jaccard_index: 0.0000e+00
256/297 [========================>.....] - ETA: 3s - loss: 0.2393 - jaccard_index: 0.0000e+00
257/297 [========================>.....] - ETA: 3s - loss: 0.2398 - jaccard_index: 0.0000e+00
258/297 [=========================>....] - ETA: 3s - loss: 0.2397 - jaccard_index: 0.0000e+00
259/297 [=========================>....] - ETA: 3s - loss: 0.2398 - jaccard_index: 0.0000e+00
260/297 [=========================>....] - ETA: 3s - loss: 0.2402 - jaccard_index: 0.0000e+00
261/297 [=========================>....] - ETA: 3s - loss: 0.2404 - jaccard_index: 0.0000e+00
262/297 [=========================>....] - ETA: 3s - loss: 0.2411 - jaccard_index: 0.0000e+00
263/297 [=========================>....] - ETA: 3s - loss: 0.2412 - jaccard_index: 0.0000e+00
264/297 [=========================>....] - ETA: 3s - loss: 0.2412 - jaccard_index: 0.0000e+00
265/297 [=========================>....] - ETA: 2s - loss: 0.2409 - jaccard_index: 0.0000e+00
266/297 [=========================>....] - ETA: 2s - loss: 0.2413 - jaccard_index: 0.0000e+00
267/297 [=========================>....] - ETA: 2s - loss: 0.2411 - jaccard_index: 0.0000e+00
268/297 [==========================>...] - ETA: 2s - loss: 0.2410 - jaccard_index: 0.0000e+00
269/297 [==========================>...] - ETA: 2s - loss: 0.2408 - jaccard_index: 0.0000e+00
270/297 [==========================>...] - ETA: 2s - loss: 0.2407 - jaccard_index: 0.0000e+00
271/297 [==========================>...] - ETA: 2s - loss: 0.2403 - jaccard_index: 0.0000e+00
272/297 [==========================>...] - ETA: 2s - loss: 0.2401 - jaccard_index: 0.0000e+00
273/297 [==========================>...] - ETA: 2s - loss: 0.2398 - jaccard_index: 0.0000e+00
274/297 [==========================>...] - ETA: 2s - loss: 0.2402 - jaccard_index: 0.0000e+00
275/297 [==========================>...] - ETA: 2s - loss: 0.2403 - jaccard_index: 0.0000e+00
276/297 [==========================>...] - ETA: 1s - loss: 0.2401 - jaccard_index: 0.0000e+00
277/297 [==========================>...] - ETA: 1s - loss: 0.2400 - jaccard_index: 0.0000e+00
278/297 [===========================>..] - ETA: 1s - loss: 0.2399 - jaccard_index: 0.0000e+00
279/297 [===========================>..] - ETA: 1s - loss: 0.2400 - jaccard_index: 0.0000e+00
280/297 [===========================>..] - ETA: 1s - loss: 0.2401 - jaccard_index: 0.0000e+00
281/297 [===========================>..] - ETA: 1s - loss: 0.2403 - jaccard_index: 0.0000e+00
282/297 [===========================>..] - ETA: 1s - loss: 0.2401 - jaccard_index: 0.0000e+00
283/297 [===========================>..] - ETA: 1s - loss: 0.2397 - jaccard_index: 0.0000e+00
284/297 [===========================>..] - ETA: 1s - loss: 0.2392 - jaccard_index: 0.0000e+00
285/297 [===========================>..] - ETA: 1s - loss: 0.2397 - jaccard_index: 0.0000e+00
286/297 [===========================>..] - ETA: 1s - loss: 0.2394 - jaccard_index: 0.0000e+00
287/297 [===========================>..] - ETA: 0s - loss: 0.2393 - jaccard_index: 0.0000e+00
288/297 [============================>.] - ETA: 0s - loss: 0.2395 - jaccard_index: 0.0000e+00
289/297 [============================>.] - ETA: 0s - loss: 0.2392 - jaccard_index: 0.0000e+00
290/297 [============================>.] - ETA: 0s - loss: 0.2390 - jaccard_index: 0.0000e+00
291/297 [============================>.] - ETA: 0s - loss: 0.2394 - jaccard_index: 0.0000e+00
292/297 [============================>.] - ETA: 0s - loss: 0.2394 - jaccard_index: 0.0000e+00
293/297 [============================>.] - ETA: 0s - loss: 0.2395 - jaccard_index: 0.0000e+00
294/297 [============================>.] - ETA: 0s - loss: 0.2400 - jaccard_index: 0.0000e+00
295/297 [============================>.] - ETA: 0s - loss: 0.2399 - jaccard_index: 0.0000e+00
296/297 [============================>.] - ETA: 0s - loss: 0.2400 - jaccard_index: 0.0000e+00
297/297 [==============================] - ETA: 0s - loss: 0.2399 - jaccard_index: 0.0000e+00
Epoch 00002: val_loss improved from 0.25942 to 0.25855, saving model to content\output\reproduce_lucchi_pp_10\checkpoints\model_weights_reproduce_lucchi_pp_10_10.h5
297/297 [==============================] - 29s 96ms/step - loss: 0.2399 - jaccard_index: 0.0000e+00 - val_loss: 0.2585 - val_jaccard_index: 0.0000e+00
Hello,
How many epochs did you run? Can you attach here the YAML config file used and the commit you are using? As we are under continuous development of BiaPy.
Thanks,
Dani
Thanks for your response Dani, the jaccard_index starts to grow after training about 150 epochs. Here are the metrics at the end of epoch 420: loss: 0.0824 - jaccard_index: 0.5985 - val_loss: 0.1672 - val_jaccard_index: 0.4564.
Below is the YAML file. Regards
AUGMENTOR:
ENABLE: true
HFLIP: true
RANDOM_ROT: true
SHUFFLE_TRAIN_DATA_EACH_EPOCH: true
SHUFFLE_VAL_DATA_EACH_EPOCH: true
VFLIP: true
DATA:
CHECK_GENERATORS: false
PATCH_SIZE: (256, 256, 1)
TEST:
IN_MEMORY: true
LOAD_GT: true
MASK_PATH: content\data\lucchi_pp\test\y
PADDING: (32, 32)
PATH: content\data\lucchi_pp\test\x
TRAIN:
IN_MEMORY: true
MASK_PATH: content\data\lucchi_pp\train\y
PATH: content\data\lucchi_pp\train\x
VAL:
FROM_TRAIN: true
SPLIT_TRAIN: 0.1
MODEL:
ARCHITECTURE: attention_unet
DROPOUT_VALUES:
- 0.1
- 0.1
- 0.2
- 0.2
- 0.3
FEATURE_MAPS: - 16
- 32
- 64
- 128
- 256
LOAD_CHECKPOINT: false
PROBLEM:
NDIM: 2D
TYPE: SEMANTIC_SEG
SYSTEM:
NUM_CPUS: 1
TEST:
ANALIZE_2D_IMGS_AS_3D_STACK: true
AUGMENTATION: true
ENABLE: true
POST_PROCESSING:
YZ_FILTERING: true
STATS:
FULL_IMG: true
MERGE_PATCHES: true
PER_PATCH: true
TRAIN:
BATCH_SIZE: 6
ENABLE: true
EPOCHS: 420
LR: 0.002
OPTIMIZER: SGD
PATIENCE: 200
Hello,
We need to check each workflow's notebook but I have just run the 2D Semantic Segmentation one with its default options, i.e. with Attention U-Net, and it's learning smoothly from the first epochs. Will update a minor things of the notebook but you can use this one in the meanwhile:
https://drive.google.com/file/d/1elOzvPxTFBDg-iEiCISvRyquqbaISxus/view?usp=sharing
Thank you for sharing the notebook, Dani! I think it is my data preprocessing issue, the notebook works fine with the Lucchi dataset but not the Lucchi++ dataset. Regards