self.compute_metrics in the kmoe_trainer

Where is the self.compute_metrics in the kmoe_trainer.py?

        if self.compute_metrics is not None and preds is not None and label_ids is not None:
            metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
        else:
            metrics = {}

Line 294.

MoKGE/trainers/kgmoe_trainer.py

Line 294 in b36a690

    
           if self.compute_metrics is not None and preds is not None and label_ids is not None:

Thanks for instant reply. With your example script for eg dataset: scripts/KGMixtureOfExpertCho.sh,

self.compute_metrics is None all the time right? For instance, in the script you pointed out, I still cannnot figure out how you assign self.compute_metrics. Can you show us how to compute the metric?

I am just wondering on your task for eg, how you compute the scores with 3 reference sentences.

Because preds and label_ids are both of size (B x 60)

Good question!

I did modify a little bit since the self.compute_metric is predefined by Huggingface Seq2Seq models. Since the prediction size is different from the input size, I add the evaluation script in

MoKGE/trainers/seq2seq_trainer.py

Line 338 in b36a690

    
           metrics.update(eval_accuracy_diversity(out_pred_path, val_ref_path, self.args.eval_beams))

and did not use the self.compute_metrics

I see, it's in the training loop, not the prediction loop.