test dataset evaluation seems to be stuck

[Bekbolatov]

Using csv iterator for cxxnet and getting this behavior.
Prediction output is a same fixed value for all instances, trying different parameters, doesn't seem to change. Notice test-error seems to be stuck.

Initializing layer: 0
Initializing layer: 1
Initializing layer: 2
Initializing layer: 3
Initializing layer: 4
Initializing layer: 5
Initializing layer: 6
Initializing layer: 7
SGDUpdater: eta=0.000100, mom=0.900000
SGDUpdater: eta=0.000100, mom=0.900000
SGDUpdater: eta=0.000100, mom=0.900000
SGDUpdater: eta=0.000100, mom=0.900000
SGDUpdater: eta=0.000100, mom=0.900000
SGDUpdater: eta=0.000100, mom=0.900000
node[in].shape: 100,1,1,111
node[!node-after-0].shape: 100,1,1,120
node[!node-after-1].shape: 100,1,1,120
node[!node-after-2].shape: 100,1,1,120
node[!node-after-3].shape: 100,1,1,120
node[!node-after-4].shape: 100,1,1,10
CSVIterator:filename=./data/train.csv
CSVIterator:filename=./data/cv.csv
initializing end, start working
round        0:[     459] 2 sec elapsed[1]      train-error:0.710458    test-error:0.630196
round        1:[     459] 4 sec elapsed[2]      train-error:0.633115    test-error:0.630196
round        2:[     459] 6 sec elapsed[3]      train-error:0.635599    test-error:0.630196

updating end, 6 sec in all

Head of conf:

  4 data = train
  5 iter = csv
  6   filename = "./data/train.csv"
  7   has_header=0
  8 iter = end
  9
 10 eval = test
 11 iter = csv
 12   filename = "./data/cv.csv"
 13   has_header=0
 14 iter = end
 15
 16
 17 netconfig=start
 18 layer[+0] = dropout
 19   threshold = 0.3
 20 layer[+1] = fullc
 21   nhidden = 120
 22 layer[+1] = sigmoid
 23 layer[+0] = dropout
 24   threshold = 0.2
 25 layer[+1] = fullc
 26   nhidden = 120
 27 layer[+1] = sigmoid
 28 layer[+1] = fullc
 29   nhidden = 10
 30 layer[+0] = softmax
 31 netconfig=end
 32
 33 # evaluation metric
 34 metric = error

For more info:

$ head -n 2 data/cv.csv
4,-0.139726696238,0.823637576747,1.04291510186,-1.39641481168,0.215977467294,-0.670459065089,0.443409886412,0.121333843115,-1.49619845661,0.0647228626288,1.11626094983,-0.164560315406,-0.204113081178,-1.31624478485,-0.358019388013,-0.807510638621,-0.177399499116,-0.708583931203,-0.0713067006171,-0.0943517528148,-0.10383585531,-0.657139421785,-0.165772761459,-0.371913589251,-0.0686180139329,2.50648167326,-0.103546069062,-0.0910159693816,-0.150530287793,0.295632746295,-0.167698336419,-0.178455125071,0.341233725387,-0.182967790591,-0.12283028663,-0.0408592935743,-0.101889042194,-0.195584399503,-0.0613127271594,-0.0673076787795,-0.236666640419,-0.460821844849,-0.126879879082,-0.167821890491,-0.231729038119,-0.060664924098,10.2161555739,-0.0898035385449,-0.473219646698,-0.222709909157,-0.436409037264,-0.120422966536,-0.159419879246,-0.216181716866,-0.0546750421818,-0.095299083493,-0.0840795496422,-0.442126515381,0.488211814488,-0.488211814488,-0.569870667891,-0.324883877602,-0.236528019189,-0.115292111661,-0.110390971179,1.01531361628,-0.145058207485,-0.270617561051,-0.521370528139,-0.264516444313,-0.426369318496,-0.0541312000975,-0.00990205792089,2.00772897419,-0.303329701054,-0.0608275120604,-0.619313336234,-0.00990205792089,0.886738252556,-0.886738252556,-0.104028616759,0.250754353544,-0.08252825731,-0.208567737916,-0.143784764004,0.315166006014,-0.176869613408,-0.203500035384,-0.238646574498,-0.0703278588075,1.06654406437,-0.853411370042,-0.223774783904,-0.0655205632715,-0.616919886357,0.616919886357,0.472133056255,-0.472133056255,-0.501843725055,-0.100902140478,-0.415459331776,-0.332755679371,0.919519653324,-1.44915655052,1.44915655052,-1.40463846262,1.88496037578,-0.330168802681,-0.101001246932,-0.0902462460218,-0.0159678412156
4,1.60179549007,-1.41432784854,-0.10693847994,-1.11826954035,-1.93573865978,1.64737864237,-0.152345252605,-1.34207142919,0.770926512596,0.0647228626288,0.0878037355797,-0.164560315406,0.889572539508,-0.26178094912,-0.358019388013,-0.807510638621,-0.177399499116,-0.708583931203,-0.0713067006171,-0.0943517528148,-0.10383585531,1.5217470857,-0.165772761459,-0.371913589251,-0.0686180139329,-0.398965614098,-0.103546069062,-0.0910159693816,6.64318134685,-3.38257521377,-0.167698336419,-0.178455125071,-2.93054269142,-0.182967790591,-0.12283028663,-0.0408592935743,-0.101889042194,-0.195584399503,16.3098274425,-0.0673076787795,-0.236666640419,-0.460821844849,-0.126879879082,-0.167821890491,-0.231729038119,-0.060664924098,-0.0978841789128,-0.0898035385449,-0.473219646698,-0.222709909157,-0.436409037264,-0.120422966536,-0.159419879246,4.62573808043,-0.0546750421818,-0.095299083493,-0.0840795496422,-0.442126515381,0.488211814488,-0.488211814488,-0.569870667891,-0.324883877602,4.22782892034,-0.115292111661,-0.110390971179,-0.984917353577,-0.145058207485,-0.270617561051,-0.521370528139,-0.264516444313,-0.426369318496,-0.0541312000975,-0.00990205792089,-0.498075194836,-0.303329701054,-0.0608275120604,1.61469153253,-0.00990205792089,0.886738252556,-0.886738252556,-0.104028616759,0.250754353544,-0.08252825731,-0.208567737916,-0.143784764004,0.315166006014,-0.176869613408,-0.203500035384,-0.238646574498,-0.0703278588075,-0.937607768308,1.17176784269,-0.223774783904,-0.0655205632715,-0.616919886357,0.616919886357,0.472133056255,-0.472133056255,-0.501843725055,-0.100902140478,-0.415459331776,-0.332755679371,0.919519653324,0.690056570934,-0.690056570934,0.711926966696,-0.530515130637,-0.330168802681,-0.101001246932,-0.0902462460218,-0.0159678412156

$ wc -l data/cv.csv
    5100 data/cv.csv

[Bekbolatov]

I got it to work - batching was not supported in this iterator.