Score column doesn't populate when reviewing trained dialogs
Closed this issue · 4 comments
Not sure if this is the right repo to report this issue. I am playing around with the Conversation Learner and creating new dialogs. When I train new dialogs and press Score Actions, I see the scores pop up. However, if I am reviewing a trained dialog, all of the scores show up as "Unknown". Seems unhelpful.
Much thanks!
-s
Hi Szymon - thanks for raising this. I agree showing a column of "unknown" is not helpful. We are undecided on whether to show scores when reviewing a train dialog. The main issue is that the model used for scoring will (almost always) have been trained on the train dialog, and this will lead to scores that imply more confidence/accuracy than we'd expect for unseen data. In other words, showing scores might be misleading. (We'd also need an API and schema change, but that's a lower-order bit). I think the safest path forward now is to hide this column -- I'll add a (internal) task for that, and update this issue when that's done. Thanks.
Hi Jason. Good feedback. I would have preferred to see the actual scores. Just so I understand your point, if we show the scores, they will be biased towards what we have trained said dialog. Right? I get that being a bit misleading. The counter argument, I suppose, is what I would like to see the scores to know whether any further training is affecting the trained dialogs negatively. I suppose I'm thinking something along the lines of LUIS where the LUIS UI allows us to see each utterance's Intents scores. I understand this is a different model, but I felt that user interface feature was effective at figuring out when utterances conflicted, etc.
That is a good point about using the scores in training dialogs to detect changes resulting from updating the training set -- eg which actions are becoming harder/easier for the to predict. I can see how that could be useful. I'll add "show scores to train dialogs" to our backlog. Thanks!
Closing for now. The train dialogs currently don't have score data available during review, but log dialogs do. Short term solution was to hide the column when reviewing train dialogs to avoid confusion of them all saying "unknown". Long term will be to get updated scores when you view the dialog and possibly compare them with the existing scores.