Model Interaction Code

Create virtual environment

setup the virtual environment

conda create -n msqa python=3.10
conda activate msqa
git clone https://github.com/ModelInteraction/MSQA.git
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch
pip install -r requirements.txt

Process pretrain Azure documentation

direct to pretrain_azure_doc/ and run the below commandline to download the azure documentation for pretrain
```
chmod +x clone_repos.sh
./clone_repos.sh
```
extract and rename markdown files and save to pretrain_azure_doc/data/
```
python save_azure.py
```
split the markwdown files into json file limited with max token length for pretrain, save json file in to pretrain_azure_doc/azure_json_output/
```
python process_azure.py
```

Process MSQA data

Note we only show sample MSQA data and full data will be available when the paper is accepted.

direct to msqa_process/
post process the msqa data collected from Microsoft Q&A forum
```
python post_process.py
```
split and save to train and test json, they should be saved to msqa_process/data/MSQA_train.json and msqa_process/data/MSQA_test.json, respectively.
```
python split.py
```

Pretrain and finetune

direct to train/

pretrain with Azure documentation following the commandline with DeepSpeed

deepspeed train.py \
--model_name_or_path {YOUR_MODEL_PATH} \
--data_path {AZURE_JSON_PATH} \
--output_dir {PRETRAIN_MODEL_SAVE_PATH} \
--num_train_epochs 8 \
--model_max_length 512 \
--per_device_train_batch_size 64 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 500 \
--save_total_limit 4 \
--learning_rate 2e-5 \
--warmup_steps 2 \
--logging_steps 2 \
--lr_scheduler_type "cosine" \
--report_to "tensorboard" \
--gradient_checkpointing True \
--deepspeed configs/deepspeed_config.json \
--fp16 True

where {AZURE_JSON_PATH} is the path where you save processed azure documentation json pretrain_azure_doc/azure_json_output/

finetune with MSQA train data previously saved in msqa_process/data/MSQA_train.json

deepspeed train.py \
--model_name_or_path {PRETRAIN_MODEL_SAVE_PATH} \
--data_path {MSQA_TRAIN_JSON_PATH} \
--output_dir {FINETUNE_MODEL_SAVE_PATH} \
--num_train_epochs 5 \
--model_max_length 512 \
--per_device_train_batch_size 64 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 20 \
--save_total_limit 3 \
--learning_rate 1e-5 \
--warmup_steps 2 \
--logging_steps 2 \
--lr_scheduler_type "cosine" \
--report_to "tensorboard" \
--gradient_checkpointing True \
--deepspeed configs/deepspeed_config.json \
--fp16 True

Inference with finetuned model

generate domain knowledge with our finetuned model with the commandline

python inference.py \
--base_model= {FINETUNE_MODEL_SAVE_PATH} \
--infer_ids_path= {QUESTION_ID_TO_INFERENCE} \
--save_path= {RESULT_SAVE_PATH} \
--batch_size=1 \
--max_new_tokens=512 \
--num_beams=4

Result generation and evaluation

Once the domain-specific model output its response to the question, we perform LLM generation taking either our domain knowledge or the chunks from retrieval-based methods.
You should save your OAI key in the keybook.py and the endpoint function of LLM is in llm_components.py.
Standard metrics, including BLEU, ROUGE-1/2/L, METEOR, BERT-Score, SIM, are defined in eval_metrics.py.
Our proposed metrics
- CAR is defined in is_no_answer in eval_metrics.py.
- KHR is defined in KHR.py and keywords need to be extracted beforehand with keyword_extract.py.
- LLM-based metrics is defined in llm_eval.py.
result_generation.py contains all prompts to generate baseline results given either domain knowledge from our model or chunks from retrieval-based methods.
score_conflict.py and conflict_stat_plot.py is to get the conflict analysis from LLM-based metric and visualization, respectively.

Human evaluation UI

We also include the UI for human evaluators

Direct to ui/

Setup the python virtual environment

conda create -n humaneval python=3.10
conda activate humaneval
pip install -r requirements.txt

put the data to be evaluated in ui/human_eval_data/
prepare the data
```
python preprocess_human_eval_data.py
```
Run the UI
```
streamlit run qa_preference.py
```

Human evaluation analysis

Direct to human_annotataion/

put the .csv files of each human evaluator to human_annotation/data/
process the human evaluation
```
python annotation_process.py
```
output statistics and plot results in annotation_stats.py

DhiMalo/MSQA