macabdul9/CASA-Dialogue-Act-Classifier

What accuracy can you actually achieve?

Opened this issue · 3 comments

xiajinxiong commented

Hi! I just cloned and ran your code but after half a day's trainig the accuracy on dev set just achieved 67%, way beyond what was reported in the original paper, which was 82% on test set.
Also, when I reproduced the model with my own code and data proprocessing techniques, the best I can achieve with hierarchical GRU was around 76% on dev set and 74% on test set.
Is the accuracy reported in the paper truly reproducable? Has anyone spotted the same issue with me?
Thanks

glicerico commented

I reached a max accuracy as you found. Please see #8 (comment)

xiajinxiong commented

@glicerico Thanks. Can you find a way to reach higher accuracy? My Roberta with transformer Encoder reached 79.9% on dev set, but was not a fair comparison with this model.

glicerico commented

I'm not the author of this repository. No, I explored a few options but didn't reach more than what I report in the comment above. Good luck!

  • 👍1