dpressel/rude-carnie

custom training error

ssetty opened this issue · 5 comments

When I try to give images of an aged person -> response comes in mid-age range < 50 years should have been 70 years. I attempted to custom train 'inception' model with images of 87 years but fails with exception. I am not clear about "fold_dir", " data_dir". Kindly help

below are commands
python preproc.py --fold_dir
/home/setsat2011/software/agegender/AgeGenderDeepLearning/Folds/train_val_txt_files_per_fold/test_fold_is_0 --train_list lata.txt --valid_list lata.txt --data_dir /home/setsat2011/software/images --output_dir /home/setsat2011/age_test_fold_is_0

python train.py --train_dir /home/setsat2011/age_test_fold_is_0 --max_steps 15000 --model_type inception --batch_size 32 --eta 0.001 --dropout 0.5 --pre_model /home/setsat2011/software/inception/inception_v3.ckpt

Caused by op 'cross_entropy_per_example/cross_entropy_per_example', defined at:
File "train.py", line 192, in
tf.app.run()
File "/home/setsat2011/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 122, in main
total_loss = loss(logits, labels)
File "train.py", line 93, in loss
logits=logits, labels=labels, name='cross_entropy_per_example')
File "/home/setsat2011/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1693, in spar
se_softmax_cross_entropy_with_logits
precise_logits, labels, name=name)
File "/home/setsat2011/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2491, in
_sparse_softmax_cross_entropy_with_logits
features=features, labels=labels, name=name)
File "/home/setsat2011/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line
767, in apply_op
op_def=op_def)
File "/home/setsat2011/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in c
reate_op
original_op=self.default_original_op, op_def=op_def)
File "/home/setsat2011/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in _
init

self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Received a label value of 87 which is outside the valid range of [0
, 1). Label values: 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 87 8
7
[[Node: cross_entropy_per_example/cross_entropy_per_example = SparseSoftmaxCrossEntropyWithLogits[T=DT_FLO
AT, Tlabels=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](output/output, batch_processing/Reshape_1)]]
setsat2011@

Check your md.json file. It looks like preproc.py found only 2 classes, not 88

This probably means whatever training data you are using is messed up

Thanks dpressel.
Please send me instructions for custom training. I need to train aged people > 70 years.
Regards

Please help/ any hint or suggestion I can try. Thanks

I do not have enough information to help you with your specific problem. In general, you need 1) training data covering the labels you wish to classify, and 2) you need to update the last layer of this network to support the range of labels you want.

IMO there is no substitute for reading and understanding the code, its not particularly complicated. Once you get whats going on, its fairly simple to make this modification, assuming (again) you have training data to cover this.