/bgi-llm-promptcblue

Primary LanguagePython

bgi-llm-promptcblue

我们参考run_clm.py，实现了对causal language模型的微调 并在CCKS2023-PromptCBLUE中文医疗大模型评测基准—开源赛道中取得了A榜第2, B榜第2的成绩。

Training procedure

我们在baichuan-13b模型基座上，对全量参数进行了有监督的微调, (区别于通用的sft方案，我们在计算loss按照预训练的策略对于全部的token均计算了loss)

https://huggingface.co/yourui/bgi-promptcblue-baichuan-13b

选择了step=50000的checkpoints作为最终的模型(max_steps=58920)

微调：

chmod 755 ./promptcblue/supervised_finetuning/fintune.sh
./promptcblue/supervised_finetuning/fintune.sh

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 1
  • eval_batch_size: 8
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 8
  • total_train_batch_size: 8
  • total_eval_batch_size: 64
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • num_epochs: 2.0

Framework versions

  • Transformers 4.30.2
  • Pytorch 2.0.1+cu118
  • Datasets 2.12.0
  • Tokenizers 0.13.3

Data

在PromptCBLUE的基础训练数据上，扩充了到了235k条训练数据, 具体扩充方法见PromptCBLUE_data 训练数据文件：file

数据: total: 166779

type 训练数据
train.json 68900
CMeEE-V2 15000
CMeIE 14291
CHIP-CDN 6000
CHIP-CDEE 1587
IMCS-V2-NER 41765
CHIP-MDCFNPC 0
IMCS-V2-SR 0
IMCS-V2-DAC 0
CHIP-CTC 22962
CHIP-STS 16000
KUAKE-IR 10000
KUAKE-QIC 5000
KUAKE-QQR 0
KUAKE-QTR 24174
MedDG 10000
IMCS-V2-MRG 0

prompt处理如下：

f"Write a response that appropriately completes the Input.\n\nInput:\n{input}\n\nResponse:\n{target}{LLAMA_EOS_TOKEN}"

Generate

下载模型https://huggingface.co/yourui/bgi-promptcblue-baichuan-13b，并保存在model目录下

为了加速推理，推理数据分成八份，每份由一张卡推理。

chmod 755 ./script/PromptCBLUE_generate/generate_all.sh 
chmod 755 ./script/PromptCBLUE_generate/baichuan/generate.sh

./script/PromptCBLUE_generate/generate_all.sh baichuan