Adding languages

[peregilk]

I am working on evaluating the performance of a minority Language (Norwegian - not included in XTREME). Unfortunately there are no good benchmarks available in the target language and we are building our own benchmarks. Such single-language benchmarks will not be useful for comparing against for instance English.

Could translating XTREME (or some of the tests) be an alternative for benchmarking individual languages as well? How much work should be estimated for doing such a job?

[melvinjosej]

Hi @peregilk,

Thanks for reaching out to us. Yes, definitely translating some of the XTREME tasks into other languages is a good way to extend the benchmark. We've thought about doing this but haven't gotten around to it. If you and your team are interested in doing this for Norwegian, we'd be happy to talk about integrating it back into XTREME.