neulab/awesome-align

Can I use the XML-R model?

Closed this issue · 5 comments

sadanyh commented

Hi
Thank you awesome-align team for the nice tool and the demo. I am trying to use XML-R with your model instead of M-BERT. I get an error with this line:
out_src = model(ids_src.unsqueeze(0), output_hidden_states=True)[2][align_layer][0, 1:-1]

My guess is that the hidden_states and model output for XML-R is different. I think the index for the word embeddings in XML-R is index [-1]. Could you please give your direction on how to solve this, if it is possible?

Thanks

zdou0830 commented

Hi, right now the XLM-R-related code is not included in this repo as we found that it underperformed mBERT as in the paper. I'll upload the XLM-R code to a different branch.

sujoysarkarcs commented

Hi, thanks for the awesome tool. Any update on using XLM-R Code? In which branch can I find it?

zdou0830 commented

Hi @sujoysarkarcs, sorry for the delay. I'll upload the code by 08/16.

sujoycs commented

Thank you.