the training is not convergent
Lvhhhh opened this issue · 12 comments
i follow the training step in the paper
firstly,i want to train the L6.but i found it not converge.
the solver.prototxt is :
THIS IS ONLY AN EXAMPLE. YOU CAN CHANGE THE SETTINGS, IF NECESSARY.
net: "train.prototxt"
base_lr: 1e-4
lr_policy: "multistep"
gamma: 0.5
TRAIN Batch size: 8
TEST Batch size: 4
test_iter: 160
test_interval: 5000
max_iter: 300000
snapshot: 30000
momentum: 0.9
weight_decay: 0.0004
display: 250
snapshot_prefix: "L6"
solver_mode: GPU
solver_type: ADAM
momentum2: 0.999
the train.prototxt is:
#######################################
LiteFlowNet CVPR 2018
by
T.-W. Hui, CUHK
#######################################
layer {
name: "CustomData1"
type: "CustomData"
top: "blob0"
top: "blob1"
top: "blob2y"
include {
phase: TRAIN
}
data_param {
# THIS IS ONLY AN EXAMPLE. YOU CAN CHANGE THE SETTINGS, IF NECESSARY.
source: "/data/chair_lmdb"
batch_size: 6
backend: LMDB
rand_permute: true
rand_permute_seed: 77
slice_point: 3
slice_point: 6
encoding: UINT8
encoding: UINT8
encoding: UINT16FLOW
verbose: true
}
}
layer {
name: "CustomData2"
type: "CustomData"
top: "blob0"
top: "blob1"
top: "blob2y"
include {
phase: TEST
}
data_param {
# THIS IS ONLY AN EXAMPLE. YOU CAN CHANGE THE SETTINGS, IF NECESSARY.
source: "/data/chair_lmdb"
batch_size: 1
backend: LMDB
rand_permute: true
rand_permute_seed: 77
slice_point: 3
slice_point: 6
encoding: UINT8
encoding: UINT8
encoding: UINT16FLOW
verbose: true
}
}
layer {
name: "Eltwise1"
type: "Eltwise"
bottom: "blob2y"
top: "blob2"
eltwise_param {
operation: SUM
coeff: 0.01
}
}
#######################################
Pre-processing
#######################################
layer {
name: "Eltwise1"
type: "Eltwise"
bottom: "blob0"
top: "blob4"
eltwise_param {
operation: SUM
coeff: 0.00392156862745
}
}
layer {
name: "Eltwise2"
type: "Eltwise"
bottom: "blob1"
top: "blob5"
eltwise_param {
operation: SUM
coeff: 0.00392156862745
}
}
layer {
name: "img0s_aug"
type: "DataAugmentation"
bottom: "blob4"
top: "img0_aug"
top: "blob7"
propagate_down: false
augmentation_param {
max_multiplier: 1
augment_during_test: false
recompute_mean: 3000
mean_per_pixel: false
translate {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.4
prob: 1.0
}
rotate {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.4
prob: 1.0
}
zoom {
rand_type: "uniform_bernoulli"
exp: true
mean: 0.2
spread: 0.4
prob: 1.0
}
squeeze {
rand_type: "uniform_bernoulli"
exp: true
mean: 0
spread: 0.3
prob: 1.0
}
lmult_pow {
rand_type: "uniform_bernoulli"
exp: true
mean: -0.2
spread: 0.4
prob: 1.0
}
lmult_mult {
rand_type: "uniform_bernoulli"
exp: true
mean: 0.0
spread: 0.4
prob: 1.0
}
lmult_add {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
sat_pow {
rand_type: "uniform_bernoulli"
exp: true
mean: 0
spread: 0.4
prob: 1.0
}
sat_mult {
rand_type: "uniform_bernoulli"
exp: true
mean: -0.3
spread: 0.5
prob: 1.0
}
sat_add {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
col_pow {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.4
prob: 1.0
}
col_mult {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.2
prob: 1.0
}
col_add {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.02
prob: 1.0
}
ladd_pow {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.4
prob: 1.0
}
ladd_mult {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0.0
spread: 0.4
prob: 1.0
}
ladd_add {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.04
prob: 1.0
}
col_rotate {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 1
prob: 1.0
}
crop_width: 448
crop_height: 320
chromatic_eigvec: 0.51
chromatic_eigvec: 0.56
chromatic_eigvec: 0.65
chromatic_eigvec: 0.79
chromatic_eigvec: 0.01
chromatic_eigvec: -0.62
chromatic_eigvec: 0.35
chromatic_eigvec: -0.83
chromatic_eigvec: 0.44
}
}
layer {
name: "aug_params1"
type: "GenerateAugmentationParameters"
bottom: "blob7"
bottom: "blob4"
bottom: "img0_aug"
top: "blob8"
augmentation_param {
augment_during_test: false
translate {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
rotate {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
zoom {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.03
prob: 1.0
}
gamma {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
brightness {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.02
prob: 1.0
}
contrast {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
color {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
}
coeff_schedule_param {
half_life: 50000
initial_coeff: 0.5
final_coeff: 1
}
}
layer {
name: "img1s_aug"
type: "DataAugmentation"
bottom: "blob5"
bottom: "blob8"
top: "img1_aug"
propagate_down: false
propagate_down: false
augmentation_param {
max_multiplier: 1
augment_during_test: false
recompute_mean: 3000
mean_per_pixel: false
crop_width: 448
crop_height: 320
chromatic_eigvec: 0.51
chromatic_eigvec: 0.56
chromatic_eigvec: 0.65
chromatic_eigvec: 0.79
chromatic_eigvec: 0.01
chromatic_eigvec: -0.62
chromatic_eigvec: 0.35
chromatic_eigvec: -0.83
chromatic_eigvec: 0.44
}
}
layer {
name: "FlowAugmentation1"
type: "FlowAugmentation"
bottom: "blob2"
bottom: "blob7"
bottom: "blob8"
top: "flow_gt_aug"
augmentation_param {
crop_width: 448
crop_height: 320
}
}
layer {
name: "FlowScaling"
type: "Eltwise"
bottom: "flow_gt_aug"
top: "scaled_flow_gt_aug"
eltwise_param {
operation: SUM
coeff: 0.05
}
}
#######################################
NetC
#######################################
layer {
name: "conv1"
type: "Convolution"
bottom: "img0_aug"
bottom: "img1_aug"
top: "F0_L1"
top: "F1_L1"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 32
pad: 3
kernel_size: 7
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU1a"
type: "ReLU"
bottom: "F0_L1"
top: "F0_L1"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU1b"
type: "ReLU"
bottom: "F1_L1"
top: "F1_L1"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv2_1"
type: "Convolution"
bottom: "F0_L1"
bottom: "F1_L1"
top: "F0_1_L2"
top: "F1_1_L2"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 32
pad: 1
kernel_size: 3
stride: 2
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU2a_1"
type: "ReLU"
bottom: "F0_1_L2"
top: "F0_1_L2"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU2b_1"
type: "ReLU"
bottom: "F1_1_L2"
top: "F1_1_L2"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv2_2"
type: "Convolution"
bottom: "F0_1_L2"
bottom: "F1_1_L2"
top: "F0_2_L2"
top: "F1_2_L2"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 32
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU2a_2"
type: "ReLU"
bottom: "F0_2_L2"
top: "F0_2_L2"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU2b_2"
type: "ReLU"
bottom: "F1_2_L2"
top: "F1_2_L2"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv2_3"
type: "Convolution"
bottom: "F0_2_L2"
bottom: "F1_2_L2"
top: "F0_L2"
top: "F1_L2"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 32
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU2a_3"
type: "ReLU"
bottom: "F0_L2"
top: "F0_L2"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU2b_3"
type: "ReLU"
bottom: "F1_L2"
top: "F1_L2"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv3_1"
type: "Convolution"
bottom: "F0_L2"
bottom: "F1_L2"
top: "F0_1_L3"
top: "F1_1_L3"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 64
pad: 1
kernel_size: 3
stride: 2
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU3a_1"
type: "ReLU"
bottom: "F0_1_L3"
top: "F0_1_L3"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU3b_1"
type: "ReLU"
bottom: "F1_1_L3"
top: "F1_1_L3"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv3_2"
type: "Convolution"
bottom: "F0_1_L3"
bottom: "F1_1_L3"
top: "F0_L3"
top: "F1_L3"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 64
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU3a_2"
type: "ReLU"
bottom: "F0_L3"
top: "F0_L3"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU3b_2"
type: "ReLU"
bottom: "F1_L3"
top: "F1_L3"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv4_1"
type: "Convolution"
bottom: "F0_L3"
bottom: "F1_L3"
top: "F0_1_L4"
top: "F1_1_L4"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 96
pad: 1
kernel_size: 3
stride: 2
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU4a_1"
type: "ReLU"
bottom: "F0_1_L4"
top: "F0_1_L4"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU4b_1"
type: "ReLU"
bottom: "F1_1_L4"
top: "F1_1_L4"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv4_2"
type: "Convolution"
bottom: "F0_1_L4"
bottom: "F1_1_L4"
top: "F0_L4"
top: "F1_L4"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 96
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU4a_2"
type: "ReLU"
bottom: "F0_L4"
top: "F0_L4"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU4b_2"
type: "ReLU"
bottom: "F1_L4"
top: "F1_L4"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv5"
type: "Convolution"
bottom: "F0_L4"
bottom: "F1_L4"
top: "F0_L5"
top: "F1_L5"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 128
pad: 1
kernel_size: 3
stride: 2
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU5a"
type: "ReLU"
bottom: "F0_L5"
top: "F0_L5"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU5b"
type: "ReLU"
bottom: "F1_L5"
top: "F1_L5"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv6"
type: "Convolution"
bottom: "F0_L5"
bottom: "F1_L5"
top: "F0_L6"
top: "F1_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 192
pad: 1
kernel_size: 3
stride: 2
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU6a"
type: "ReLU"
bottom: "F0_L6"
top: "F0_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "ReLU6b"
type: "ReLU"
bottom: "F1_L6"
top: "F1_L6"
relu_param { negative_slope: 0.1 }
}
#######################################
NetE-M: L6
#######################################
layer {
name: "corr_L6"
type: "Correlation"
bottom: "F0_L6"
bottom: "F1_L6"
top: "corr_L6"
correlation_param {
pad: 3
kernel_size: 1
max_displacement: 3
stride_1: 1
stride_2: 1
}
}
layer {
name: "ReLU_corr_L6"
type: "ReLU"
bottom: "corr_L6"
top: "corr_L6"
relu_param {
negative_slope: 0.1
}
}
layer {
name: "conv1_D1_L6"
type: "Convolution"
bottom: "corr_L6"
top: "conv1_D1_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 128
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU1_D1_L6"
type: "ReLU"
bottom: "conv1_D1_L6"
top: "conv1_D1_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv2_D1_L6"
type: "Convolution"
bottom: "conv1_D1_L6"
top: "conv2_D1_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 64
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU2_D1_L6"
type: "ReLU"
bottom: "conv2_D1_L6"
top: "conv2_D1_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv3_D1_L6"
type: "Convolution"
bottom: "conv2_D1_L6"
top: "conv3_D1_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 32
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU3_D1_L6"
type: "ReLU"
bottom: "conv3_D1_L6"
top: "conv3_D1_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "scaled_flow_D1_L6"
type: "Convolution"
bottom: "conv3_D1_L6"
top: "scaled_flow_D1_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 2
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "Downsample_L6"
type: "Downsample"
bottom: "scaled_flow_gt_aug"
bottom: "scaled_flow_D1_L6"
top: "scaled_flow_label_L6"
propagate_down: false
propagate_down: false
}
layer {
name: "scaled_flow_D1_L6_loss"
type: "L1Loss"
bottom: "scaled_flow_D1_L6"
bottom: "scaled_flow_label_L6"
top: "scaled_flow_D1_L6_loss"
loss_weight: 0.32
l1_loss_param { l2_per_location: true }
}
#######################################
NetE-S: L6
#######################################
layer {
name: "FlowUnscaling_L6_D2"
type: "Eltwise"
bottom: "scaled_flow_D1_L6"
top: "flow_D1_L6"
eltwise_param {
operation: SUM
coeff: 0.625
}
}
layer {
name: "gxy_L6"
type: "Grid"
top: "gxy_L6"
bottom: "flow_D1_L6"
propagate_down: false
}
layer {
name: "coords_D1_L6"
type: "Eltwise"
bottom: "flow_D1_L6"
bottom: "gxy_L6"
top: "coords_D1_L6"
eltwise_param { coeff: 1 coeff: 1 }
}
layer {
name: "warped_F1_L6"
type: "Warp"
bottom: "F1_L6"
bottom: "coords_D1_L6"
top: "warped_D1_F1_L6"
}
layer {
name: "F_D2_L6"
bottom: "F0_L6"
bottom: "warped_D1_F1_L6"
bottom: "scaled_flow_D1_L6"
top: "F_D2_L6"
type: "Concat"
concat_param { axis: 1 }
}
layer {
name: "conv1_D2_L6"
type: "Convolution"
bottom: "F_D2_L6"
top: "conv1_D2_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 128
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU1_D2_L6"
type: "ReLU"
bottom: "conv1_D2_L6"
top: "conv1_D2_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv2_D2_L6"
type: "Convolution"
bottom: "conv1_D2_L6"
top: "conv2_D2_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 64
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU2_D2_L6"
type: "ReLU"
bottom: "conv2_D2_L6"
top: "conv2_D2_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv3_D2_L6"
type: "Convolution"
bottom: "conv2_D2_L6"
top: "conv3_D2_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 32
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "ReLU3_D2_L6"
type: "ReLU"
bottom: "conv3_D2_L6"
top: "conv3_D2_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "scaled_flow_D2_res_L6"
type: "Convolution"
bottom: "conv3_D2_L6"
top: "scaled_flow_D2_res_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 2
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "scaled_flow_D2_L6"
type: "Eltwise"
bottom: "scaled_flow_D1_L6"
bottom: "scaled_flow_D2_res_L6"
top: "scaled_flow_D2_L6"
eltwise_param { operation: SUM }
}
layer {
name: "scaled_flow_D2_L6_loss"
type: "L1Loss"
bottom: "scaled_flow_D2_L6"
bottom: "scaled_flow_label_L6"
top: "scaled_flow_D2_L6_loss"
loss_weight: 0.32
l1_loss_param { l2_per_location: true }
}
#######################################
NetE-R: L6
#######################################
layer {
name: "slice_scaled_flow_D_L6"
type: "Slice"
bottom: "scaled_flow_D2_L6"
top: "scaled_flow_D_L6_x"
top: "scaled_flow_D_L6_y"
slice_param { axis: 1 slice_point: 1 }
}
layer {
name: "reshaped_scaled_flow_D_L6_x"
type: "Im2col"
bottom: "scaled_flow_D_L6_x"
top: "reshaped_scaled_flow_D_L6_x"
convolution_param { pad: 1 kernel_size: 3 stride: 1 }
}
layer {
name: "reshaped_scaled_flow_D_L6_y"
type: "Im2col"
bottom: "scaled_flow_D_L6_y"
top: "reshaped_scaled_flow_D_L6_y"
convolution_param { pad: 1 kernel_size: 3 stride: 1 }
}
layer {
name: "mean_scaled_flow_D_L6_x"
type: "Reduction"
bottom: "scaled_flow_D_L6_x"
top: "mean_scaled_flow_D_L6_x"
reduction_param { operation: MEAN axis: 1 coeff: -1 }
}
layer {
name: "scaled_flow_D_nomean_L6_x"
type: "Bias"
bottom: "scaled_flow_D_L6_x"
bottom: "mean_scaled_flow_D_L6_x"
top: "scaled_flow_D_nomean_L6_x"
bias_param { axis: 0 }
}
layer {
name: "mean_scaled_flow_D_L6_y"
type: "Reduction"
bottom: "scaled_flow_D_L6_y"
top: "mean_scaled_flow_D_L6_y"
reduction_param { operation: MEAN axis: 1 coeff: -1 }
}
layer {
name: "scaled_flow_D_nomean_L6_y"
type: "Bias"
bottom: "scaled_flow_D_L6_y"
bottom: "mean_scaled_flow_D_L6_y"
top: "scaled_flow_D_nomean_L6_y"
bias_param { axis: 0 }
}
layer {
name: "FlowUnscaling_L6_R"
type: "Eltwise"
bottom: "scaled_flow_D2_L6"
top: "flow_D2_L6"
eltwise_param {
operation: SUM
coeff: 0.625
}
}
layer {
name: "Downsample_img0_aug_L6"
type: "Downsample"
bottom: "img0_aug"
bottom: "flow_D2_L6"
top: "img0_aug_L6"
propagate_down: false
propagate_down: false
}
layer {
name: "Downsample_img1_aug_L6"
type: "Downsample"
bottom: "img1_aug"
bottom: "flow_D2_L6"
top: "img1_aug_L6"
propagate_down: false
propagate_down: false
}
layer {
name: "coords_R_L6"
type: "Eltwise"
bottom: "flow_D2_L6"
bottom: "gxy_L6"
top: "coords_R_L6"
eltwise_param { coeff: 1 coeff: 1 }
}
layer {
name: "warped_img1_aug_L6"
type: "Warp"
bottom: "img1_aug_L6"
bottom: "coords_R_L6"
top: "warped_img1_aug_L6"
}
layer {
name: "img_diff_L6"
type: "Eltwise"
bottom: "img0_aug_L6"
bottom: "warped_img1_aug_L6"
top: "img_diff_L6"
eltwise_param {
operation: SUM
coeff: 1.0
coeff: -1.0
}
}
layer {
name: "channelNorm_L6"
type: "ChannelNorm"
bottom: "img_diff_L6"
top: "channelNorm_L6"
}
layer {
name: "concat_F0_R_L6"
type: "Concat"
bottom: "channelNorm_L6"
bottom: "scaled_flow_D_nomean_L6_x"
bottom: "scaled_flow_D_nomean_L6_y"
bottom: "F0_L6"
top: "concat_F0_R_L6"
concat_param { axis: 1 }
}
layer {
name: "conv1_R_L6"
type: "Convolution"
bottom: "concat_F0_R_L6"
top: "conv1_R_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 128
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "relu_conv1_R_L6"
type: "ReLU"
bottom: "conv1_R_L6"
top: "conv1_R_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv2_R_L6"
type: "Convolution"
bottom: "conv1_R_L6"
top: "conv2_R_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 128
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "relu_conv2_R_L6"
type: "ReLU"
bottom: "conv2_R_L6"
top: "conv2_R_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv3_R_L6"
type: "Convolution"
bottom: "conv2_R_L6"
top: "conv3_R_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 64
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "relu_conv3_R_L6"
type: "ReLU"
bottom: "conv3_R_L6"
top: "conv3_R_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv4_R_L6"
type: "Convolution"
bottom: "conv3_R_L6"
top: "conv4_R_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 64
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "relu_conv4_R_L6"
type: "ReLU"
bottom: "conv4_R_L6"
top: "conv4_R_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv5_R_L6"
type: "Convolution"
bottom: "conv4_R_L6"
top: "conv5_R_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 32
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "relu_conv5_R_L6"
type: "ReLU"
bottom: "conv5_R_L6"
top: "conv5_R_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "conv6_R_L6"
type: "Convolution"
bottom: "conv5_R_L6"
top: "conv6_R_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 1 decay_mult: 0 }
convolution_param {
num_output: 32
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" }
engine: CUDNN
}
}
layer {
name: "relu_conv6_R_L6"
type: "ReLU"
bottom: "conv6_R_L6"
top: "conv6_R_L6"
relu_param { negative_slope: 0.1 }
}
layer {
name: "dist_R_L6"
type: "Convolution"
bottom: "conv6_R_L6"
top: "dist_R_L6"
param { lr_mult: 1 decay_mult: 1 }
param { lr_mult: 0 }
convolution_param {
num_output: 9
pad: 1
kernel_size: 3
stride: 1
weight_filler { type: "msra" }
bias_filler { type: "constant" value: 0 }
engine: CUDNN
}
}
layer {
name: "sq_dist_R_L6"
type: "Power"
bottom: "dist_R_L6"
top: "sq_dist_R_L6"
power_param { power: 2 scale: 1 shift: 0 }
}
layer {
name: "neg_sq_dist_R_L6"
type: "Eltwise"
bottom: "sq_dist_R_L6"
top: "neg_sq_dist_R_L6"
eltwise_param { operation: SUM coeff: -1 }
}
layer {
name: "exp_kernel_R_L6"
type: "Softmax"
bottom: "neg_sq_dist_R_L6"
top: "exp_kernel_R_L6"
softmax_param { axis: 1 engine: CUDNN }
}
layer {
name: "f-lconv_L6_x"
type: "Eltwise"
bottom: "reshaped_scaled_flow_D_L6_x"
bottom: "exp_kernel_R_L6"
top: "f-lconv_L6_x"
eltwise_param { operation: PROD }
}
layer {
name: "scaled_flow_R_L6_x"
type: "Convolution"
bottom: "f-lconv_L6_x"
top: "scaled_flow_R_L6_x"
param { lr_mult: 0 }
param { lr_mult: 0 }
convolution_param {
num_output: 1
kernel_size: 1
weight_filler { type: "constant" value: 1 }
}
}
layer {
name: "f-lconv_L6_y"
type: "Eltwise"
bottom: "reshaped_scaled_flow_D_L6_y"
bottom: "exp_kernel_R_L6"
top: "f-lconv_L6_y"
eltwise_param { operation: PROD }
}
layer {
name: "scaled_flow_R_L6_y"
type: "Convolution"
bottom: "f-lconv_L6_y"
top: "scaled_flow_R_L6_y"
param { lr_mult: 0 }
param { lr_mult: 0 }
convolution_param {
num_output: 1
kernel_size: 1
weight_filler { type: "constant" value: 1 }
}
}
layer {
name: "scaled_flow_R_L6"
bottom: "scaled_flow_R_L6_x"
bottom: "scaled_flow_R_L6_y"
top: "scaled_flow_R_L6"
type: "Concat"
concat_param { axis: 1 }
}
layer {
name: "scaled_flow_R_L6_loss"
type: "L1Loss"
bottom: "scaled_flow_R_L6"
bottom: "scaled_flow_label_L6"
top: "scaled_flow_R_L6_loss"
loss_weight: 0.32
l1_loss_param { l2_per_location: true }
}
This sounds strange. The training can be converged on my side. I would suggest you to send me the training log instead of posting the prototxt files here.
I am not sure whether your training and testing sets are correct or not. This could be one of the reasons.
Did you pre-train the matching and sub-pixel refinement networks before adding the regularization network? The training procedure can be found in my CVPR18 paper.
this is the log of pre-train the matching and sub-pixel refinement networks of L6
it still not converge
i have check the lmdb is correct
420.log
Did you modify any parameters in my solver.prototxt and train.prototxt?
I find that you modified my solver.prototxt and train.prototxt.
- You did not specify stepvalue in your solver.prototxt. My template has this parameter!!
- You multiplied 0.01 to blob2. I did NOT do so!!
Before you make any complaints, please make sure you did not modify any important parameters in my codes. This is really unfair to me.
I find that you modified my solver.prototxt and train.prototxt.
- You did not specify stepvalue in your solver.prototxt. My template has this parameter!!
- You multiplied 0.01 to blob2. I did NOT do so!!
Before you make any complaints, please make sure you did not modify any important parameters in my codes. This is really unfair to me.
- in Training Details. i found the first step is 1)NetC and M6:S6 of NetE is trained for 300k iterations, and Learning rates are initially set to 1e-4. so i just want to realise this step and add others after that. no need to do this?
- i multiply 100 during making lmdb . so the value of blob2 multiplied 0.01 is equal to the value in *.flo and i test it correctly~
As mentioned clearly in my paper and the provided solver.prototxt, multi-step learning rate is used.
As mentioned clearly in my paper and the provided solver.prototxt, multi-step learning rate is used.
fineonly using the train.prototxt.template and solver.prototxt.template you provided, it cannot converge as well. later i will show you the log
fine..only using the train.prototxt.template and solver.prototxt.template you provided, it cannot converge as well. later i will show you the log
when you make the lmdb. do you multi the flow vlaue ? for example:
// Read flow
for (int j = 0; j<2widthheight; j++) {
short value=0;
if(has_flow())
{
if(isnan(flo_data[j]))
value=std::numeric_limits::max();
else
value=flo_data[j] * 100;* //(multiply 100 and multiplt 0.01 in train.prototxt to make the precision of flow value to 0.01)
}
*(ptr++)=*((unsigned char*)&value);
*(ptr++)=*((unsigned char*)&value+1);
}
assert(ptr==data+10*width*height);
There is no need to modify anything in Caffe source files. You over-do it!