error rate is high in the eval mode

[maxh2018]

I'm not sure if I didn't understand your code. According to the terminal commands you showed, I got an error rate of up to 49% in validation mode

[cxy1997]

Are the pretrained weights successfully loaded? You should see Loading pretrained weight from ... in the log if so.
If you see a line that says Creating a new model., your checkpoint path might be incorrect.