the problem of scripts

[wuhongyan123]

Hello, I would like to ask if the framework supports directly specifying parameters for training and validation sets? I see in args.py that there is a train_file and validation_file parameter but the command line running the script doesn't recognize the two parameters. If I use the dataset_path provided by the script, how do I determine which is the optimal one for all the checkpoints saved at the end?

[research4pan]

Thanks for your interest in LMFlow! To specify validation files, you may try the following options,

...
--do_eval \
--eval_dataset_path ./data/val/ \
--evaluation_strategy steps \
--eval_steps 100 \
--save_steps 100 \

This will result in the evaluation of every checkpoint, with the corresponding validation loss in wandb or log outputs. You can determine the best checkpoint based on metrics there. Hope that answers your question 😄