/multilingual-modeling-1

Primary LanguagePythonApache License 2.0Apache-2.0

README

Installation

  1. Install the packages from composable-sft.
  2. Install the packages from rational_activations. You would need to follow the [Other CUDA/PyTorch] section for installation.
  3. Uninstall transformers using the command pip uninstall transformers.
  4. Install the packages from this repo using pip install -r requirements.txt.

Language Adaptations

See the scripts in lang_adapt/scripts/run_clm_* as examples for the following language adaptation strategies:

  • Adaptable adapters
  • BitFit
  • Continual Pretraining
  • IA3
  • LoRA
  • MAD-X
  • Pfeiffer
  • Pretraining from Scratch
  • Composable SFT

WikiANN

See the scripts in scripts/eval/scripts_wikiann/pilot_*_.sh as examples for evaluating the adapted models on WikiANN.