Model Name
conceptofmind opened this issue · 3 comments
conceptofmind commented
@lucidrains What would you like the first model to be named?
conceptofmind commented
One restart. 160B tokens.
conceptofmind commented
Did some tests with qk_norm vs no qk_norm as well. When using AdamW decided to go with qk_norm=False. I will explore this with Lion after.
conceptofmind commented
PaLM 1B