Evaluate v0.1.0 wrt. the validation set
Opened this issue · 1 comments
ninpnin commented
Evaluate this model: https://github.com/welfare-state-analytics/bert-riksdagen-classifier/releases/tag/v0.1.0 . We probably want a separate script that takes in a huggingface model, and evaluates it on a (validation) dataset.
ninpnin commented
The new release (v0.2.0) works with the huggingface pipeline
>>> from transformers import pipeline
>>> cls = pipeline("text-classification", model="./ft2classes")
>>> cls("Herr talman!")
[{'label': 'note', 'score': 0.6480845212936401}]
Now we just need to adjust the script to actually use the hugginface class, so that we can evaluate both models v0.1 and v0.2