Other Languages

[pepeballesterostel]

I would like to know if this module is able to adapt to other languagges such as Spanish, to obtain metrics generated from a finetuned GPT-2.

Thanks!

[juharris]

Yes it should work but some of the metrics we use might need to be configured or just ignored. See #82 #14. Let us know how it goes and if you have any feedback for us.

[pepeballesterostel]

The metrics worked fine for spanish text. But I want to use Spanish models for a better evaluation.
For instance, to use a Spanish version of Glove, Should I change this line of code (line 93 in nlg-eval/bin/nlg-eval):
url='https://raw.githubusercontent.com/manasRK/glovegensim/42ce46f00e83d3afa028fb6bf17ed3c90ca65fcc/glove2word2vec.py',

And change it to an alternative Spanish model url? Should I change lines 102 and 105 from the same script?

Thank you.

[kracwarlock]

The glove2word2vec.py file converts the glove embeddings into word2vec format so that should still be usable. You would need to change the glove files from English to Spanish so you are right about changing lines 102 and 105:

nlg-eval/bin/nlg-eval

Lines 102 to 107 in 7f79930

	setup_glove = not os.path.exists(os.path.join(data_path, 'glove.6B.300d.model.bin'))
	if setup_glove:
	downloads.append(dict(
	url='http://nlp.stanford.edu/data/glove.6B.zip',
	target_dir=data_path
	))

Another thing is that we use NLTK's word_tokenize tokenizer:

nlg-eval/nlgeval/word2vec/evaluate.py

Line 47 in 7f79930

from nltk.tokenize import word_tokenize

nlg-eval/nlgeval/word2vec/evaluate.py

Line 63 in 7f79930

embs = [emb.vec(word) for word in word_tokenize(hyp)]

To a limited extent it will work for Spanish but it would be better to also replace that with a Spanish tokenizer. If you are planning to publish this work somewhere, it would also be good to let your readers know which tokenizer you ended up using for comparability.

If you are planning to use other non-embedding metrics from this repo then maybe for those as well a Spanish tokenizer would be more appropriate.

[juharris]

Closing due to inactivity and the question is mostly answered.