As the guthub has limititation on the file sizes, we use the zenodo to save the whole replication package. Please check this link (https://zenodo.org/record/7339088#.Y3oBonZBzIU) to download the complete repo.
- Python3.8+
- CUDA Version: 11.7
- Conda
Note: The CUDA Version needs to be 11.7 to ensure compatibility and functionality.
$ conda create -n llm4correct python=3.8
$ conda activate llm4correct
This step may take several minutes:
$ conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
$ bash install_library.sh
Please run the following pipeline script:
$ bash run_pipeline.sh
Notes:
- '--task' the format is Patch_{APR_Tool_Name} and it is to choose the target APR tool, e.g. Patch_ACS
- '--option' is to choose the guiding information for LLM. "bug-trace-testcase-similar" is the default parameter.
- the content in the '--out_dir' is the logits generated by LLM.
- the default '--max_length' is 4000 while if you meet OOM problem, you can reduce it accordingly.
- the default '--batch_size' is 1 while if you have extra memory, you can set to 2 to speed up.
After finishing all inferences, you can run this Python file to read the results for each APR tool:
$ python read_results.py
Notes:
- Need to revise the path to the '--out_dir' in the last step
This also includes implementations of many recent papers studying in-context learning.
- Brown et al. NeurIPS 2021. "Language Models are Few-Shot Learners"
- Zhao et al. ICML 2021. "Calibrate before use: Improving few-shot performance of language models"
- Holzman et al. EMNLP 2021. "Surface Form Competition: Why the Highest Probability Answer Isn't Always Right"
- Sewon et al. 2022. "Noisy Channel Language Model Prompting for Few-Shot Text Classification"