llama2-7B AssertionError: padded_vocab_size value from checkpoint (32000) is not equal to the input argument value (32256) #81

[yushengsu-thu]

I follow here and use the same arguemnts:
https://epfllm.github.io/Megatron-LLM/guide/getting_started.html

When I training,

LOG_ARGS="--log_interval 1 --save_interval 100 --eval_interval 50"
TRAIN_ARGS="--train_iters 6500 --lr_decay_style cosine --lr_warmup_iters 650 --lr 2e-5 --min_lr 2e-6"
DISTRIBUTED_ARGS="--nproc_per_node $NUMBER_OF_GPUS_for_EACH_NODE --nnodes $NUMBER_OF_NODES --node_rank $NODE_ID --master_addr localhost --master_port 6000"
torchrun $DISTRIBUTED_ARGS ../finetune.py
--tensor_model_parallel_size 2
--pipeline_model_parallel_size 1
--load $LLM_LOAD_DIR
--save $LLM_SAVE_DIR
--tensorboard_dir $TENSORBOARD_DIR
--data_path $DATA_DIR
--model_name llama2
--tokenizer_type SentencePieceTokenizer
--vocab_file $VOCAB_PREFIX/tokenizer.model
--bf16
--use_flash_attn
--micro_batch_size 8
--global_batch_size 64
--sequence_parallel
--recompute_granularity selective
--use_checkpoint_args
--data_type instruction
--variable_seq_lengths
--vocab_extra_ids_list "<|im_start|>,<|im_end|>"
$COMMON_ARGS $LOG_ARGS $TRAIN_ARGS $LLAMA_ARGS

I encountered the following problem:
"llama2-7B AssertionError: padded_vocab_size value from checkpoint (32000) is not equal to the input argument value (32256) #81"

When I shared the model, I used 32000 true_vocab_size as you said in the torrential link (I also tried to remove it) but I still encounter the same error.

VOCAB_SIZE=32000
python3 ../tools/checkpoint_util.py
--target_tensor_parallel_size 2
--target_pipeline_parallel_size 1
--load_dir $LLM_LOAD_DIR
--save_dir $LLM_SAVE_SHARDED_DIR
--model_type llama2
--true_vocab_size $VOCAB_SIZE
--bf16

[Gannn12138]

I have the same problem

[LinglingGreat]

add --no_new_tokens to your args