MantisAI/nervaluate

List of possible formats

ivyleavedtoadflax opened this issue · 11 comments

ivyleavedtoadflax commented

What other formats do we want the package to be able to use? For starters:

  • list of labels
  • Prodigy
  • spaCy
  • StanfordNER
  • CoNLLa
  • Single-Lineb
  • xml: TODO (add example)
  • inlineXML: TODO (add example)
  • tsv: TODO (add example)
  • slashTags: TODO (add example)
  • a
,	O
Davos	PERSON
2018	O
:	O
Soros	PERSON
accuses	O
Trump	PERSON
of	O
wanting	O
a	O
`	O
mafia	O
state	O
'	O
and	O
blasts	O
social	O
media	O
.	O
  • b
Switzerland/LOCATION ,/O Davos/PERSON 2018/O :/O Soros/PERSON accuses/O Trump/PERSON of/O wanting/O a/O /O mafia/O state/O '/O and/O blasts/O social/O media/O ./O`
ivyleavedtoadflax commented

There is an implementation of CoNLL to spacy here: explosion/spaCy#533 (comment) which should be easy to adapt to the prodigy format now used by nervaluate.

ivyleavedtoadflax commented

@davidsbatista I've merged this into #3 so will close this here for now.

ivyleavedtoadflax commented

Ok - will close the correct issue this time!

gunturbudi commented

I would love to see the Flair output compatible with this library.

https://github.com/flairNLP/flair/blob/master/resources/docs/TUTORIAL_2_TAGGING.md

  • 👍1
ivyleavedtoadflax commented

Thanks @gunturbudi for the suggestion. I'd love to know more if you are currently using this package and what your use cases might be. We are just kicking off some development on it.

gunturbudi commented

Hi, we are conducting an experiment on NER for informal medical phrases. Since Flair library did not provide a complete evaluation, we really glad we found this package :)

ivyleavedtoadflax commented

That's great to hear @gunturbudi, and I'm very glad you find it useful.

Just to make you aware, there are a few bugs which are listed in the other issues which will affect the evaluation in some cases. We've just this week started to work on the package again, so I'm hoping that we will resolve those issues soon. But just to make you aware 👀

gunturbudi commented

Thanks for the information @ivyleavedtoadflax
We also cross-check it with the evaluation from flair. So far the difference is quite small. But we will wait and see for the update :)

ivyleavedtoadflax commented

Thanks for the information @ivyleavedtoadflax
We also cross-check it with the evaluation from flair. So far the difference is quite small. But we will wait and see for the update :)

If you don't mind sharing the differences you've found with flair, that could be useful.