astramind-ai/Mixture-of-depths
Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
Python
Issues
- 2
Training error
#9 opened by Mi5sssss - 1
Gradient propagation
#10 opened by NickNickGo - 1
Question about the impl
#11 opened by GeneZC - 0
Deepspeed is Not Supported
#12 opened by PhoebusSi - 3
qwen1.5出现越界
#8 opened by FB-wh - 4
Training scripts for the MoD models
#6 opened by Aafiya-H - 6
- 2
recalculate capacity question
#4 opened by starsholic - 1
training question on LLama_factory
#2 opened by putizi-super - 1
Can we combine this method with moe?
#3 opened by kostum123 - 1
Misaligned implementation with paper
#5 opened by starsholic - 2
training question
#1 opened by bobzhang208