Problem with Wav2Vec finetune task

[ayw001]

Last year with fairseq, I successfully finetuned both base and large model with my own data. The result was very good.
However, I run into the problem when I try to repeat the same finetune task.
At the beginning, the training seems normal as eval_loss decline as expected but eval_wer is still 1.0. But eventual, valid_loss go to 'nan'
and the output model does not produce anything.
One thing I observed in log file is
ctc_loss_reduction": "sum"
According to your blog, that should be "mean".

I do not configure that at all.

Is this a problem? If it is, how do I fix it.

Thanks

Yuanjun