Eval logic

[faad3]

Hello! I'm trying to figure out the logic. As far as I understand, there are two (three) options in the config for the eval key:
1.1) specify 'pure' without injectors and then step.do_forward_backward will be performed
1.2) specify 'pure' with injectors, and then they will be executed
2) specify evaluators

Do I understand correctly that (1.2) differs from (2) in that there may be more complex logic in the evaluator than in the injector?

Anyway, I found a config (recipes/vqvae2/train_imgnet_vqvae_stage1.yml) in which the eval key has neither the pure key nor the evaluators key, but the injectors key right away. How will it work in this case?

[neonbjb]

Pure=true will use the training losses for evaluation, which is an effective way of monitoring for overfitting a lot of the time and requires no effort on your part.

Frankly, i dont manually specify eval injectors anymore. I would not be surprised if that option was broken in some way and i dont recommend using it unless you want to do some testing.

The evaluator path is my recommendation if you want a custom eval metric like fid, accuracy, wer, etc. There are a bunch of sample evaluators available for you to reference (or use). This is very important for diffusion and gan models where the training losses are completely disconnected from downstream quality.

If you dont specify an evaluator or set pure=false, the trainer will just not do evaluation passes.

[faad3]

Thanks!