Code for the paper "Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect"
List of models released for this paper:
- hf.co/ZurichNLP/swissbert
- this paper adds a Swiss German adapter
- the original version of SwissBERT is accessible at model revision v1)
- hf.co/ZurichNLP/swiss-german-xlm-roberta-base
- hf.co/ZurichNLP/swiss-german-canine
- hf.co/ZurichNLP/swiss-german-swissbert-char
- Requirements: Python >= 3.8, PyTorch
pip install -r requirements.txt
- Not all the data we used are publicly available. See
data/README.md
for details. python -m scripts.preprocess_continued_pretraining_data
- Subword level:
python -m scripts.continued_pretraining_subword <model_name_or_path>
- Tested with
xlm-roberta-base
,facebook/xmod-base
,ZurichNLP/swissbert
- Tested with
- Character level:
python -m scripts.continued_pretraining_char <model_name_or_path>
- Tested with
google/canine-s
,facebook/xmod-base
,ZurichNLP/swissbert
(the latter two correspond to the GLOBI approach described in Section 4.3 of the paper)
- Tested with
- See
data/README.md
for instructions on how to download the data.
- Part-of-speech tagging:
python -m scripts.evaluate_pos <model_name_or_path>
- German dialect identification:
python -m scripts.evaluate_gdi <model_name_or_path>
- Retrieval (no fine-tuning):
python -m scripts.evaluate_retrieval <model_name_or_path>
- This code repository: MIT license
- Model weights: Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)
@inproceedings{vamvas-etal-2024-modular,
title={Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect},
author={Jannis Vamvas and No{\"e}mi Aepli and Rico Sennrich},
booktitle={First Workshop on Modular and Open Multilingual NLP},
year={2024},
}