Workshop on LLMs in Prod
- CUDA Toolkit 117
- C++ Compiler (i.e. gcc 11.3.0)
- Python >= 3.10
To install the python dependencies simply run
poetry install
In order to pull the EleutherAI/lm-evaluation-harness
and stanford-crfm/helm
you will need to run the following line:
git submodule update --init --recursive
Although you can download models using model.from_pretrained
, it's recommended to use the git-lfs
to download the models.
git-lfs install
git clone https://huggingface.co/google/flan-ul2
To launch a fine-tuning job using Deepspeed ZeRO3 (without CPU offloading) run the below command:
accelerate launch --config_file ./finetune/launcher_configs/accelerate_zero3_no_offload_config.yaml ./finetune/zero3_gsm8k.py <model-path>
In order to conduct evaluations on the GSM8K dataset run the below script.
python lm-evaluation-harness/main.py --model hf-seq2seq --model_args pretrained=google/flan-ul2,dtype=float16,use_accelerate=True --task=gsm8k --batch_size=16 --write_out --output_base_path=./eval_results/
The notebooks and code are inspired from various work from Hugging Face, EleutherAI, and stanford-crfm.