Context Window of PhoGPT ?

[xnohat]

Hi team,

As in model card have said that PhoGPT using ALiBi for context length extrapolation , so team have tested maximum effective context length of PhoGPT ?

Thanks for the first GPT foundation model for Vietnamese

[datquocnguyen]

Not yet. It would be fine if you were using two times the current maximum length of the context, i.e., 2048 * 2 = 4096: config.max_seq_len = 4096. There are a few previously existing GPT-style LLM models for Vietnamese. Please refer to our technical report.