AetherCortex/Llama-X

only load lm_head.weight and embed_tokens.weight parameters

Opened this issue · 0 comments

I can only load these parameters,
○ lm_head.weight : 131076096
○ model.embed_tokens.weight : 131076096
and the other parameter is None.

Number of parameters:  262152192
model.embed_tokens.weight : 131076096
model.layers.0.self_attn.q_proj.weight : 0
model.layers.0.self_attn.k_proj.weight : 0
model.layers.0.self_attn.v_proj.weight : 0
model.layers.0.self_attn.o_proj.weight : 0
model.layers.0.mlp.gate_proj.weight : 0
model.layers.0.mlp.down_proj.weight : 0
model.layers.0.mlp.up_proj.weight : 0
model.layers.0.input_layernorm.weight : 0
model.layers.0.post_attention_layernorm.weight : 0