Evaluate Model show some warnings
fming opened this issue · 7 comments
when I try to run code of 4.5, it show some warnings like below:
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details. WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.
How do I check if my evaluation is correct?
and also for the step 4.6 I cannot find the file score/score_map/decode.log.result.map.sys
I can make sure "examples/asr/timit/data/cmvn" is generated.
The warning message always appear when conducting decoding. It wouldn't effect decoding result and can be safely ignored.
Can you check if there are decoding results inside decode.log and sclite is installed? And if so, can you please upload your decode.log?
@Some-random Thanks for your replying, here is my decode.log
decode.log
It seems there missed a lot of information. I have installed the sclite. I suspect it is related to case #288
It's not related to cmvn files because missing cmvn file will only cause degradation in performance. If you're running the recipe with something like nohup, there is a chance the actual decoding log will be written into nohup.out instead of decode.log. You will have to take these log out and put it into decode.log manually and rerun stage 4.
decode.log should be composed of lines like these:
INFO:absl:predictions: tf.Tensor(
[[21 42 12 7 11 35 34 42 47 7 46 5 14 36 32 2 7 11 27 14 4 7 46 18
16 35 34 10 37 4 7 11 33 12 7 46 4 18 26 27 43 12 38 34 42 7 39 20
44 12 23 7 22 37 10 16 30 39 4 7 22 14 21 49]], shape=(1, 64), dtype=int64) labels: [[21 42 12 36 24 10 36 42 34 42 47 7 46 5 14 20 36 32 2 7 11 27 14 4
7 46 18 35 10 37 4 7 11 16 5 12 7 46 4 18 26 38 43 12 4 34 42 7
39 16 20 26 12 23 7 22 37 10 18 27 39 4 7 14 21]] errs: 17 avg_acc: 0.7805sec/iter: 6.8036
INFO:absl:predictions: tf.Tensor(
[[21 32 48 13 38 7 11 21 31 41 12 39 10 37 47 41 7 26 38 34 27 7 39 33
16 14 27 47 21 5 16 47 36 45 4 12 16 30 37 38 7 11 18 7 22 2 39 33
16 14 20 14 21 49]], shape=(1, 54), dtype=int64) labels: [[21 32 48 13 38 7 11 21 31 41 12 39 10 37 47 38 21 26 38 36 12 4 34 42
7 39 33 16 14 20 47 21 5 18 47 36 45 4 12 16 2 38 47 42 37 4 7 11
18 7 22 2 39 33 16 14 27 14 21]] errs: 14 avg_acc: 0.7804 sec/iter: 5.9716
INFO:absl:predictions: tf.Tensor(
[[21 26 10 7 11 14 21 13 20 16 35 34 10 35 28 1 7 46 16 5 14 4 14 27
43 21 41 44 26 10 36 24 17 38 47 16 10 14 10 36 24 27 7 22 14 7 11 4
12 14 10 44 42 12 36 32 42 14 7 11 27 36 19 35 28 27 12 21 49]], shape=(1, 69), dtype=int64) labels: [[21 26 10 7 11 14 21 13 18 16 35 34 10 35 28 1 7 46 16 5 14 4 14 27
43 41 44 26 10 7 11 17 4 47 16 10 14 10 44 36 24 27 7 22 14 7 11 4
12 14 27 44 42 12 44 4 14 7 11 27 36 19 35 28 27 12 21]] errs: 10 avg_acc: 0.7810 sec/iter: 7.2863
Please check your log file to see if they exist
decode.log.result.map.sys.txt
decode.log
@Some-random I've checked the decode.log, the lines don't exist.
My way is to run the code:
source examples/asr/timit/run_101.sh
here are some warnings in the console:
INFO:absl:predictions: tf.Tensor( [[21 32 48 13 38 7 11 21 31 41 12 39 10 37 47 4 7 21 38 12 42 7 39 33 16 14 27 47 21 5 16 47 36 45 4 12 16 2 21 38 47 10 37 4 7 11 18 7 22 2 39 33 16 14 27 14 21 49]], shape=(1, 58), dtype=int64) labels: [[21 32 48 13 38 7 11 21 31 41 12 39 10 37 47 38 21 26 38 36 12 4 34 42 7 39 33 16 14 20 47 21 5 18 47 36 45 4 12 16 2 38 47 42 37 4 7 11 18 7 22 2 39 33 16 14 27 14 21]] errs: 10 avg_acc: 0.7785 sec/iter: 7.8171 INFO:absl:predictions: tf.Tensor( [[21 26 10 7 11 14 21 13 18 16 35 34 10 35 28 1 7 46 16 5 14 4 14 27 12 41 44 26 10 7 11 17 4 47 16 10 14 10 44 36 24 27 7 22 14 7 11 4 12 14 7 11 27 36 19 35 28 27 12 21 49]], shape=(1, 61), dtype=int64) labels: [[21 26 10 7 11 14 21 13 18 16 35 34 10 35 28 1 7 46 16 5 14 4 14 27 43 41 44 26 10 7 11 17 4 47 16 10 14 10 44 36 24 27 7 22 14 7 11 4 12 14 27 44 42 12 44 4 14 7 11 27 36 19 35 28 27 12 21]] errs: 8 avg_acc: 0.7795 sec/iter: 9.7761 INFO:absl:decoding finished WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details. WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.
Also I have checked score/score_map/decode.log.result.map.sys, the values are 0.
decode.log.result.map.sys.txt
decode.log@Some-random I've checked the decode.log, the lines don't exist.
My way is to run the code:
source examples/asr/timit/run_101.sh
here are some warnings in the console:
INFO:absl:predictions: tf.Tensor( [[21 32 48 13 38 7 11 21 31 41 12 39 10 37 47 4 7 21 38 12 42 7 39 33 16 14 27 47 21 5 16 47 36 45 4 12 16 2 21 38 47 10 37 4 7 11 18 7 22 2 39 33 16 14 27 14 21 49]], shape=(1, 58), dtype=int64) labels: [[21 32 48 13 38 7 11 21 31 41 12 39 10 37 47 38 21 26 38 36 12 4 34 42 7 39 33 16 14 20 47 21 5 18 47 36 45 4 12 16 2 38 47 42 37 4 7 11 18 7 22 2 39 33 16 14 27 14 21]] errs: 10 avg_acc: 0.7785 sec/iter: 7.8171 INFO:absl:predictions: tf.Tensor( [[21 26 10 7 11 14 21 13 18 16 35 34 10 35 28 1 7 46 16 5 14 4 14 27 12 41 44 26 10 7 11 17 4 47 16 10 14 10 44 36 24 27 7 22 14 7 11 4 12 14 7 11 27 36 19 35 28 27 12 21 49]], shape=(1, 61), dtype=int64) labels: [[21 26 10 7 11 14 21 13 18 16 35 34 10 35 28 1 7 46 16 5 14 4 14 27 43 41 44 26 10 7 11 17 4 47 16 10 14 10 44 36 24 27 7 22 14 7 11 4 12 14 27 44 42 12 44 4 14 7 11 27 36 19 35 28 27 12 21]] errs: 8 avg_acc: 0.7795 sec/iter: 9.7761 INFO:absl:decoding finished WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details. WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.
Also I have checked score/score_map/decode.log.result.map.sys, the values are 0.
Hi fming,
It seems like your decode results were printing to the console instead of decode.log, which is why the values in decode.log.result.map.sys is 0. I updated our codes in order to write the inference log directly to a file via python script in this pr. Try to pull the codes and feel free to reply if you have further questions.
Thanks,
Ne
@neneluo it works after getting the latest changes.
here is result I've got from score/score_map/inference.log.result.map.sys :
Sum/Avg| 192 7215 | 84.6 11.5 3.9 3.4 18.9 99.5 |
|================================================================|
| Mean | 1.0 37.6 | 84.7 11.6 3.7 3.6 18.9 99.5 |
| S.D. | 0.0 11.7 | 7.5 6.3 3.7 4.0 9.0 7.2 |
Median | 1.0 36.0 | 85.5 11.4 2.9 2.7 17.6 100.0