German T5 Model Evaluation
This project documents the results of our evaluation of German T5 models.
Summarization Task - max_source_length 800 max_target_length: 96
Setup
- train data: Swisstext
- test data: MLSUM
- GPUs: 4 (V100)
- batch size / GPU: 2
- batch size total: 8
- warmup_ratio: 0.3
- epochs: 10
- max_source_length: 800
- max_target_length: 96
- learning rate: 5e-5 (default)
Result
Higher metric is better.
| Model | rouge1 | rouge2 | rougeL | rougeLsum |
|---|---|---|---|---|
| google/mt5-small | 16.7323 | 3.5629 | 12.65 | 14.6898 |
| philschmid/test-german-t5-prompted-germanquad | 15.7629 | 2.8154 | 11.898 | 13.9223 |
| stefan-it/t5-base-secret package3 | 15.7427 | 2.9186 | 12.0224 | 13.8726 |
| stefan-it/t5-base-secret epoch2-package2 | 15.4757 | 2.7629 | 11.978 | 13.5326 |
| GermanT5/t5-base-german-3e | 14.5525 | 2.0007 | 11.1617 | 12.9124 |
| GermanT5/t5-efficient-oscar-german-small-el32 last CP | 16.6277 | 3.404 | 12.6183 | 14.5772 |
| GermanT5/t5-efficient-oscar-german-small-el32 2nd last CP | 16.6886 | 3.4468 | 12.666 | 14.6423 |
Summarization Task - max_source_length 512 max_target_length: 96 epochs: 8
Setup
- train data: Swisstext
- test data: MLSUM
- GPUs: 4 (V100)
- batch size / GPU: 2
- batch size total: 8
- warmup_ratio: 0.3
- epochs: 8
- max_source_length: 512
- max_target_length: 96
- learning rate: 5e-5 (default)
Result
Higher metric is better.
| Model | rouge1 | rouge2 | rougeL | rougeLsum |
|---|---|---|---|---|
| google/mt5-small (FP32) | 16.0354 | 3.2689 | 12.2063 | 14.1225 |
| GermanT5/t5-efficient-oscar-german-small-el32 (FP32) | 16.2004 | 3.2372 | 12.3031 | 14.2256 |
| GermanT5/t5-efficient-gc4-german-small-el32 (FP32) | 17.1507 | 3.8038 | 13.0836 | 15.1671 |
| GermanT5/t5-efficient-gc4-german-base-nl36-old (FP16) | 9.3494 | 1.3531 | 7.4037 | 8.5473 |
GermanT5/t5-efficient-gc4-german-base-nl36 (DeepSpeed with ZeRO-3 Example auto config) |
17.9501 | 3.9247 | 13.3758 | 15.6139 |