Opencv error and Array dimension error when train cityscapes datasets from scratch
liuzhenboo opened this issue · 0 comments
Hello,I want to train cityscapes datasets from scratch, But when I run:
developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log
some opencv errors occurs:
`developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
INTERFACE:
data yaml: cfg/cityscapes/data.yaml
net yaml: cfg/cityscapes/net_bonnet.yaml
train yaml: cfg/cityscapes/train_bonnet.yaml
log dir ./log
model path None
model type iou
Commit hash (training version): b'7bf03b0'
Opening desired data file cfg/cityscapes/data.yaml
Opening desired net file cfg/cityscapes/net_bonnet.yaml
Opening desired train file cfg/cityscapes/train_bonnet.yaml
Copying files to ./log for further reference.
Training from scratch
Fetching dataset
Training with 1 GPU's
Training with batch size 2
DEVICE AVAIL: /device:CPU:0
DEVICE AVAIL: /device:GPU:0
Number of GPU's available is 1
This means 2 images per GPU
Data depth: 3
Parsing directory /shared/datasets/cityscapes/bonnet_data/train
Total number of pixels: 6239027200
Total number of pixels of class road: 2036049408
Total number of pixels of class train: 12863940
Total number of pixels of class car: 386502976
Total number of pixels of class sky: 221459584
Total number of pixels of class trafficsign: 30521320
Total number of pixels of class fence: 48487192
Total number of pixels of class sidewalk: 336031712
Total number of pixels of class truck: 14775009
Total number of pixels of class person: 67202392
Total number of pixels of class vegetation: 878734336
Total number of pixels of class pole: 67768880
Total number of pixels of class building: 1259774336
Total number of pixels of class bus: 12995807
Total number of pixels of class rider: 7444910
Total number of pixels of class bicycle: 22849672
Total number of pixels of class terrain: 63965240
Total number of pixels of class trafficlight: 11509943
Total number of pixels of class motorcycle: 5445904
Total number of pixels of class wall: 36211256
Total number of pixels of class crap: 718432320
Content percentage of class road in dataset: 0.326341
Content percentage of class car in dataset: 0.061949
Content percentage of class trafficsign in dataset: 0.004892
Content percentage of class sidewalk in dataset: 0.053860
Content percentage of class truck in dataset: 0.002368
Content percentage of class vegetation in dataset: 0.140845
Content percentage of class building in dataset: 0.201918
Content percentage of class bus in dataset: 0.002083
Content percentage of class terrain in dataset: 0.010252
Content percentage of class wall in dataset: 0.005804
Content percentage of class train in dataset: 0.002062
Content percentage of class sky in dataset: 0.035496
Content percentage of class fence in dataset: 0.007772
Content percentage of class motorcycle in dataset: 0.000873
Content percentage of class person in dataset: 0.010771
Content percentage of class pole in dataset: 0.010862
Content percentage of class bicycle in dataset: 0.003662
Content percentage of class rider in dataset: 0.001193
Content percentage of class trafficlight in dataset: 0.001845
Content percentage of class crap in dataset: 0.115151
Total amount of images: 2975
OpenCV Error: Assertion failed (dims <= 2 && step[0] > 0) in locateROI, file /io/opencv/modules/core/src/matrix.cpp, line 991
Exception in thread ImgBufftrain:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/shared/bonnet/train_py/dataset/abstract_dataset.py", line 84, in run
img, lbl = self.augment(img, lbl)
File "/shared/bonnet/train_py/dataset/abstract_dataset.py", line 68, in augment
img = cv2.blur(img,(ksize,ksize))
cv2.error: /io/opencv/modules/core/src/matrix.cpp:991: error: (-215) dims <= 2 && step[0] > 0 in function locateROI
Parsing directory /shared/datasets/cityscapes/bonnet_data/valid
Total number of pixels: 1048576000
Total number of pixels of class road: 345222080
Total number of pixels of class train: 1032100
Total number of pixels of class car: 59759312
Total number of pixels of class sky: 30708080
Total number of pixels of class trafficsign: 6110454
Total number of pixels of class fence: 7527026
Total number of pixels of class sidewalk: 49559052
Total number of pixels of class truck: 2760469
Total number of pixels of class person: 11890229
Total number of pixels of class vegetation: 158682896
Total number of pixels of class pole: 13564731
Total number of pixels of class building: 200895344
Total number of pixels of class bus: 3564221
Total number of pixels of class rider: 1970543
Total number of pixels of class bicycle: 6500852
Total number of pixels of class terrain: 7625936
Total number of pixels of class trafficlight: 1813749
Total number of pixels of class motorcycle: 728922
Total number of pixels of class wall: 6720678
Total number of pixels of class crap: 131939472
Content percentage of class road in dataset: 0.329229
Content percentage of class car in dataset: 0.056991
Content percentage of class trafficsign in dataset: 0.005827
Content percentage of class sidewalk in dataset: 0.047263
Content percentage of class truck in dataset: 0.002633
Content percentage of class vegetation in dataset: 0.151332
Content percentage of class building in dataset: 0.191589
Content percentage of class bus in dataset: 0.003399
Content percentage of class terrain in dataset: 0.007273
Content percentage of class wall in dataset: 0.006409
Content percentage of class train in dataset: 0.000984
Content percentage of class sky in dataset: 0.029286
Content percentage of class fence in dataset: 0.007178
Content percentage of class motorcycle in dataset: 0.000695
Content percentage of class person in dataset: 0.011339
Content percentage of class pole in dataset: 0.012936
Content percentage of class bicycle in dataset: 0.006200
Content percentage of class rider in dataset: 0.001879
Content percentage of class trafficlight in dataset: 0.001730
Content percentage of class crap in dataset: 0.125827
Total amount of images: 500
SPECIFIC TO CITYSCAPES
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
SPECIFIC TO CITYSCAPES
Parsing directory /shared/datasets/cityscapes/bonnet_data/test
Total number of pixels: 15250000
Total number of pixels of class road: 0
Total number of pixels of class train: 0
Total number of pixels of class car: 0
Total number of pixels of class sky: 0
Total number of pixels of class trafficsign: 0
Total number of pixels of class fence: 0
Total number of pixels of class sidewalk: 0
Total number of pixels of class truck: 0
Total number of pixels of class person: 0
Total number of pixels of class vegetation: 0
Total number of pixels of class pole: 0
Total number of pixels of class building: 0
Total number of pixels of class bus: 0
Total number of pixels of class rider: 0
Total number of pixels of class bicycle: 0
Total number of pixels of class terrain: 0
Total number of pixels of class trafficlight: 0
Total number of pixels of class motorcycle: 0
Total number of pixels of class wall: 0
Total number of pixels of class crap: 15250000
Content percentage of class road in dataset: 0.000000
Content percentage of class car in dataset: 0.000000
Content percentage of class trafficsign in dataset: 0.000000
Content percentage of class sidewalk in dataset: 0.000000
Content percentage of class truck in dataset: 0.000000
Content percentage of class vegetation in dataset: 0.000000
Content percentage of class building in dataset: 0.000000
Content percentage of class bus in dataset: 0.000000
Content percentage of class terrain in dataset: 0.000000
Content percentage of class wall in dataset: 0.000000
Content percentage of class train in dataset: 0.000000
Content percentage of class sky in dataset: 0.000000
Content percentage of class fence in dataset: 0.000000
Content percentage of class motorcycle in dataset: 0.000000
Content percentage of class person in dataset: 0.000000
Content percentage of class pole in dataset: 0.000000
Content percentage of class bicycle in dataset: 0.000000
Content percentage of class rider in dataset: 0.000000
Content percentage of class trafficlight in dataset: 0.000000
Content percentage of class crap in dataset: 1.000000
Total amount of images: 1525
SPECIFIC TO CITYSCAPES
Don't weigh the 'crap' class (key 255)
SPECIFIC TO CITYSCAPES
Successfully imported datasets
Train data samples: 2975
Validation data samples: 500
Test data samples: 1525
Initializing network
TRAINING GRAPH
Building graph
encoder
downsample1
W: [5, 5, 3, 13] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
downsample2
W: [5, 5, 16, 32] Train: True
non-bt-1
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-2
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-3
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-4
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
downsample3
W: [5, 5, 48, 32] Train: True
non-bt-1
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-2
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-3
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-4
W: [7, 1, 80, 80] Train: True
`
`Saving this iteration of training in ./log/lr_0.0001
Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)
developer@linux:/shared/bonnet/train_py$
`
I think it is because a thread is break!
so I comment augment function as below:
`
# if self.name == "ImgBufftrain":
# img, lbl = self.augment(img, lbl)
`
But! new error occurs!!!!!!!!!!! some information is below：
`Total number of parameters in network: 1871287
Reporting accuracy every 10 epochs
Saving this iteration of training in ./log/lr_0.0001
Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)
`
a detail error infomation is:
`developer@linux:/shared/bonnet/train_py$ ./cnn_train.py -d cfg/cityscapes/data.yaml -n cfg/cityscapes/net_bonnet.yaml -t cfg/cityscapes/train_bonnet.yaml -l ./log
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
中文
INTERFACE:
data yaml: cfg/cityscapes/data.yaml
net yaml: cfg/cityscapes/net_bonnet.yaml
train yaml: cfg/cityscapes/train_bonnet.yaml
log dir ./log
model path None
model type iou
Commit hash (training version): b'7bf03b0'
Opening desired data file cfg/cityscapes/data.yaml
Opening desired net file cfg/cityscapes/net_bonnet.yaml
Opening desired train file cfg/cityscapes/train_bonnet.yaml
Copying files to ./log for further reference.
Training from scratch
Fetching dataset
Training with 1 GPU's
Training with batch size 20
DEVICE AVAIL: /device:CPU:0
DEVICE AVAIL: /device:GPU:0
Number of GPU's available is 1
This means 20 images per GPU
Data depth: 3
Parsing directory /shared/datasets/cityscapes/bonnet_data/train
Total number of pixels: 6239027200
Total number of pixels of class road: 2036049408
Total number of pixels of class train: 12863940
Total number of pixels of class car: 386502976
Total number of pixels of class sky: 221459584
Total number of pixels of class trafficsign: 30521320
Total number of pixels of class fence: 48487192
Total number of pixels of class sidewalk: 336031712
Total number of pixels of class truck: 14775009
Total number of pixels of class person: 67202392
Total number of pixels of class vegetation: 878734336
Total number of pixels of class pole: 67768880
Total number of pixels of class building: 1259774336
Total number of pixels of class bus: 12995807
Total number of pixels of class rider: 7444910
Total number of pixels of class bicycle: 22849672
Total number of pixels of class terrain: 63965240
Total number of pixels of class trafficlight: 11509943
Total number of pixels of class motorcycle: 5445904
Total number of pixels of class wall: 36211256
Total number of pixels of class crap: 718432320
Content percentage of class road in dataset: 0.326341
Content percentage of class car in dataset: 0.061949
Content percentage of class trafficsign in dataset: 0.004892
Content percentage of class sidewalk in dataset: 0.053860
Content percentage of class truck in dataset: 0.002368
Content percentage of class vegetation in dataset: 0.140845
Content percentage of class building in dataset: 0.201918
Content percentage of class bus in dataset: 0.002083
Content percentage of class terrain in dataset: 0.010252
Content percentage of class wall in dataset: 0.005804
Content percentage of class train in dataset: 0.002062
Content percentage of class sky in dataset: 0.035496
Content percentage of class fence in dataset: 0.007772
Content percentage of class motorcycle in dataset: 0.000873
Content percentage of class person in dataset: 0.010771
Content percentage of class pole in dataset: 0.010862
Content percentage of class bicycle in dataset: 0.003662
Content percentage of class rider in dataset: 0.001193
Content percentage of class trafficlight in dataset: 0.001845
Content percentage of class crap in dataset: 0.115151
Total amount of images: 2975
**************************** SPECIFIC TO CITYSCAPES ****************************
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
**************************** SPECIFIC TO CITYSCAPES ****************************
imhfetcher
Parsing directory /shared/datasets/cityscapes/bonnet_data/valid
Total number of pixels: 1048576000
Total number of pixels of class road: 345222080
Total number of pixels of class train: 1032100
Total number of pixels of class car: 59759312
Total number of pixels of class sky: 30708080
Total number of pixels of class trafficsign: 6110454
Total number of pixels of class fence: 7527026
Total number of pixels of class sidewalk: 49559052
Total number of pixels of class truck: 2760469
Total number of pixels of class person: 11890229
Total number of pixels of class vegetation: 158682896
Total number of pixels of class pole: 13564731
Total number of pixels of class building: 200895344
Total number of pixels of class bus: 3564221
Total number of pixels of class rider: 1970543
Total number of pixels of class bicycle: 6500852
Total number of pixels of class terrain: 7625936
Total number of pixels of class trafficlight: 1813749
Total number of pixels of class motorcycle: 728922
Total number of pixels of class wall: 6720678
Total number of pixels of class crap: 131939472
Content percentage of class road in dataset: 0.329229
Content percentage of class car in dataset: 0.056991
Content percentage of class trafficsign in dataset: 0.005827
Content percentage of class sidewalk in dataset: 0.047263
Content percentage of class truck in dataset: 0.002633
Content percentage of class vegetation in dataset: 0.151332
Content percentage of class building in dataset: 0.191589
Content percentage of class bus in dataset: 0.003399
Content percentage of class terrain in dataset: 0.007273
Content percentage of class wall in dataset: 0.006409
Content percentage of class train in dataset: 0.000984
Content percentage of class sky in dataset: 0.029286
Content percentage of class fence in dataset: 0.007178
Content percentage of class motorcycle in dataset: 0.000695
Content percentage of class person in dataset: 0.011339
Content percentage of class pole in dataset: 0.012936
Content percentage of class bicycle in dataset: 0.006200
Content percentage of class rider in dataset: 0.001879
Content percentage of class trafficlight in dataset: 0.001730
Content percentage of class crap in dataset: 0.125827
Total amount of images: 500
**************************** SPECIFIC TO CITYSCAPES ****************************
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
**************************** SPECIFIC TO CITYSCAPES ****************************
Parsing directory /shared/datasets/cityscapes/bonnet_data/test
Total number of pixels: 15250000
Total number of pixels of class road: 0
Total number of pixels of class train: 0
Total number of pixels of class car: 0
Total number of pixels of class sky: 0
Total number of pixels of class trafficsign: 0
Total number of pixels of class fence: 0
Total number of pixels of class sidewalk: 0
Total number of pixels of class truck: 0
Total number of pixels of class person: 0
Total number of pixels of class vegetation: 0
Total number of pixels of class pole: 0
Total number of pixels of class building: 0
Total number of pixels of class bus: 0
Total number of pixels of class rider: 0
Total number of pixels of class bicycle: 0
Total number of pixels of class terrain: 0
Total number of pixels of class trafficlight: 0
Total number of pixels of class motorcycle: 0
Total number of pixels of class wall: 0
Total number of pixels of class crap: 15250000
Content percentage of class road in dataset: 0.000000
Content percentage of class car in dataset: 0.000000
Content percentage of class trafficsign in dataset: 0.000000
Content percentage of class sidewalk in dataset: 0.000000
Content percentage of class truck in dataset: 0.000000
Content percentage of class vegetation in dataset: 0.000000
Content percentage of class building in dataset: 0.000000
Content percentage of class bus in dataset: 0.000000
Content percentage of class terrain in dataset: 0.000000
Content percentage of class wall in dataset: 0.000000
Content percentage of class train in dataset: 0.000000
Content percentage of class sky in dataset: 0.000000
Content percentage of class fence in dataset: 0.000000
Content percentage of class motorcycle in dataset: 0.000000
Content percentage of class person in dataset: 0.000000
Content percentage of class pole in dataset: 0.000000
Content percentage of class bicycle in dataset: 0.000000
Content percentage of class rider in dataset: 0.000000
Content percentage of class trafficlight in dataset: 0.000000
Content percentage of class crap in dataset: 1.000000
Total amount of images: 1525
**************************** SPECIFIC TO CITYSCAPES ****************************
Don't weigh the 'crap' class (key 255)
Content percentage of class crap in dataset: inf
**************************** SPECIFIC TO CITYSCAPES ****************************
Successfully imported datasets
Train data samples: 2975
Validation data samples: 500
Test data samples: 1525
Initializing network
******************************** TRAINING GRAPH ********************************
GRAPH GPU:0
Building graph
encoder
downsample1
W: [5, 5, 3, 13] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
downsample2
W: [5, 5, 16, 32] Train: True
non-bt-1
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-2
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-3
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
non-bt-4
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
W: [5, 1, 48, 48] Train: True
W: [1, 5, 48, 48] Train: True
downsample3
W: [5, 5, 48, 32] Train: True
non-bt-1
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-2
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-3
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-4
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
godeep
non-bt-1
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-2
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-3
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
non-bt-4
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
W: [7, 1, 80, 80] Train: True
W: [1, 7, 80, 80] Train: True
============= End of encoder ===============
size of code: [20, 80, 48, 96]
=========== Beginning of decoder============
decoder
upsample
unpool1
W: [2, 2, 80, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
W: [3, 1, 48, 48] Train: True
W: [1, 3, 48, 48] Train: True
unpool2
W: [2, 2, 48, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
W: [3, 1, 32, 32] Train: True
W: [1, 3, 32, 32] Train: True
unpool3
W: [2, 2, 32, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [3, 1, 16, 16] Train: True
W: [1, 3, 16, 16] Train: True
W: [1, 1, 16, 20] Train: True
b: [20] Train: True
Defining loss function
Weights for loss function (1/log(frec(c)+e)):
[ 3.36258268 14.03326797 4.98947144 39.25167084 36.5057373 32.89967728
46.27554703 40.67154694 6.70474911 33.55268478 18.51486206 32.99533463
47.68305206 12.69611359 45.20449448 45.78203583 45.8253479 48.40744781
42.75931931 0. ]
Weight decay: 1e-06
Using tensorflow gradients
******************************** TESTING GRAPH *********************************
GRAPH GPU:0
Building graph
encoder
downsample1
W: [5, 5, 3, 13] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
downsample2
W: [5, 5, 16, 32] Train: False
non-bt-1
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
non-bt-2
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
non-bt-3
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
non-bt-4
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
W: [5, 1, 48, 48] Train: False
W: [1, 5, 48, 48] Train: False
downsample3
W: [5, 5, 48, 32] Train: False
non-bt-1
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-2
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-3
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-4
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
godeep
non-bt-1
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-2
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-3
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
non-bt-4
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
W: [7, 1, 80, 80] Train: False
W: [1, 7, 80, 80] Train: False
============= End of encoder ===============
size of code: [20, 80, 48, 96]
=========== Beginning of decoder============
decoder
upsample
unpool1
W: [2, 2, 80, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
W: [3, 1, 48, 48] Train: False
W: [1, 3, 48, 48] Train: False
unpool2
W: [2, 2, 48, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
W: [3, 1, 32, 32] Train: False
W: [1, 3, 32, 32] Train: False
unpool3
W: [2, 2, 32, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [3, 1, 16, 16] Train: False
W: [1, 3, 16, 16] Train: False
W: [1, 1, 16, 20] Train: False
b: [20] Train: False
Total number of parameters in network: 1871287
Reporting accuracy every 10 epochs
Saving this iteration of training in ./log/lr_0.0001
Training model
Training network 100000 epochs (14875000 iterations at batch size 20)
Decaying learn rate by 1.010000 every 1 epochs (148 steps)
Traceback (most recent call last):
File "./cnn_train.py", line 194, in
net.train()
File "/shared/bonnet/train_py/arch/abstract_net.py", line 1265, in train
[self.train_op, self.loss], feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1109, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
terminate called without an active exception
terminate called recursively
Aborted (core dumped)`
I hope you can help me, sincerely！