/voxpopuli

A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation

Primary LanguagePythonOtherNOASSERTION

VoxPopuli

https://aclanthology.org/2021.acl-long.80

A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.

Overview

VoxPopuli provides

  • 400K hours of unlabelled speech data for 23 languages
  • 1.8K hours of transcribed speech data for 16 languages
  • 17.3K hours of speech-to-speech interpretation data for 15x15 directions

The raw data is collected from 2009-2020 European Parliament event recordings. We acknowledge the European Parliament for creating and sharing these materials.

Detailed statistics

Unlabelled and transcribed data

Language Code Unlabelled Hours (v1/v2) Transcribed Hours Transcribed Speakers Transcribed Tokens LM Tokens
English En 4.5K/24.1K 543 1313 4.8M 60.1M
German De 4.5K/23.2K 282 531 2.3M 50.0M
French Fr 4.5K/22.8K 211 534 2.1M 58.6M
Spanish Es 4.4K/21.4K 166 305 1.6M 57.4M
Polish Pl 4.5K/21.2K 111 282 802K 13.6M
Italian It 4.6K/21.9K 91 306 757K 52.1M
Romanian Ro 4.5K/17.9K 89 164 739K 10.3M
Hungarian Hu 4.4K/17.7K 63 143 431K 13.0M
Czech Cs 4.5K/18.7K 62 138 461K 13.5M
Dutch Nl 4.5K/19.0K 53 221 488K 54.6M
Finnish Fi 4.4K/14.2K 27 84 160K 34.5M
Croatian Hr 2.7K/8.1K 43 83 337K 285K
Slovak Sk 4.4K/12.1K 35 96 270K 13.3M
Slovene Sl 4.4K/11.3K 10 45 76K 12.6M
Estonian Et 4.3K/10.6K 3 29 18K 11.3M
Lithuanian Lt 4.3K/14.4K 2 21 10K 11.5M
Portuguese Pt 4.4K/17.5K - - - -
Bulgarian Bg 4.3K/17.6K - - - -
Greek El 4.4K/17.7K - - - -
Latvian Lv 4.4K/13.1K - - - -
Maltese Mt 4.4K/9.1K - - - -
Swedish Sv 4.5K/16.3K - - - -
Danish Da 4.3K/13.6K - - - -
Total 100K/384K 1791 4295 15M 467M

Speech-to-speech interpretation data

Source/Target En De Fr Es Pl It Ro Hu Cs Nl Fi Sk Sl Lt Da Total
En - 463 427 441 432 461 457 382 427 400 442 433 434 398 370 6.0K
De 187 - 196 204 214 217 198 205 214 196 217 208 218 164 179 2.8K
Fr 169 187 - 187 172 197 195 144 170 158 168 168 156 139 134 2.3K
Es 130 138 135 - 118 148 128 93 118 115 124 114 108 83 86 1.6K
Pl 68 66 54 55 - 67 55 43 67 42 55 62 57 50 34 775
It 69 77 76 79 72 - 75 61 68 64 71 66 70 53 60 961
Ro 60 59 59 58 49 61 - 38 50 43 48 50 46 38 29 688
Hu 30 38 25 27 29 30 27 - 27 20 31 29 26 21 18 378
Cs 39 35 29 30 36 32 31 23 - 23 29 55 29 25 18 434
Nl 31 43 35 29 27 38 24 25 25 - 32 25 23 19 25 401
Fi 15 18 15 13 13 13 13 12 13 11 - 14 12 11 9 182
Hr 31 27 27 24 27 28 24 22 24 22 24 26 37 21 20 384
Sk 21 22 14 16 19 16 16 14 32 13 16 - 17 13 10 239
Sl 6 6 4 5 5 6 5 4 5 4 5 6 - 4 3 68
Lt 1 1 1 1 1 1 1 1 1 1 1 1 1 - 0 13
Total 857 1.2K 1.1K 1.2K 1.2K 1.3K 1.2K 1.1K 1.2K 1.1K 1.3K 1.3K 1.2K 1.0K 995 17.3K

Getting Data

We provide raw audios as well as scripts to segment and align them with transcription/interpretation. The output format is Ogg Vorbis (16000Hz, 16-bit, mono-channel), which is supported by common libraries such as libsndfile and libsox (they have Python frontends by soundfile, torchaudio, etc.).

As the first step, clone this repo for the processing scripts

git clone https://github.com/facebookresearch/voxpopuli.git

and install required PyPI packages:

pip install -r requirements.txt

Unlabelled Data

First, download raw audios via

python -m voxpopuli.download_audios --root [ROOT] --subset [SUBSET]

which saves audios to ${ROOT}/raw_audios/[language]/[year]/[recording_id].ogg.

SUBSET specifies the data subset to download:

--subset # Languages Hours Years Size
en, de, fr, es, pl, it, ro, hu, cs, nl, fi, hr, sk, sl, et, lt, pt, bg, el, lv, mt, sv or da 1 2.7K-4.6K 2009-2020 44G-75G
en_v2, de_v2, fr_v2, es_v2, pl_v2, it_v2, ro_v2, hu_v2, cs_v2, nl_v2, fi_v2, hr_v2, sk_v2, sl_v2, et_v2, lt_v2, pt_v2, bg_v2, el_v2, lv_v2, mt_v2, sv_v2 or da_v2 1 8.1K-24.1K 2009-2020 130G-385G
10k 23 10K 2019-2020 170G
100k 23 100K 2009-2020 1.7T
400k 23 400K 2009-2020 6.4T

Then, segment these audios via

python -m voxpopuli.get_unlabelled_data --root [ROOT] --subset [SUBSET]

which outputs to ${ROOT}/unlabelled_data/[language]/[year]/[segment_id].ogg

Transcribed (ASR) Data

First, download raw audios via

python -m voxpopuli.download_audios --root [ROOT] --subset asr

which saves audios to ${ROOT}/raw_audios/original/[year]/[recording_id].ogg.

Then, segment these audios and align them with transcripts via

python -m voxpopuli.get_asr_data --root [ROOT] --lang [LANGUAGE]

which outputs

  • audios ${ROOT}/transcribed_data/[language]/[year]/[segment_id].ogg
  • per-split manifest (ID, transcript, speaker ID) ${ROOT}/transcribed_data/[language]/asr_[split].tsv

Speech-to-Speech Interpretation Data

First, follow the instructions above to set up ASR data (source audios and transcripts).

Then, download target audios via

python -m voxpopuli.download_audios --root [ROOT] --subset [TARGET_LANGUAGE]

which saves audios to ${ROOT}/raw_audios/[target_language]/[year]/[recording_id].ogg.

Finally, segment these audios and match them with source ones via

python -m voxpopuli.get_s2s_data --root [ROOT] --source-lang [SOURCE_LANGUAGE] --target-lang [TARGET_LANGUAGE]

which outputs

  • target audios ${ROOT}/transcribed_data/[language]/[target_language]/[year]/[segment_id].ogg
  • manifest (source ID, transcript, speaker ID, target ID) ${ROOT}/transcribed_data/[language]/[target_language]/s2s.tsv

We also human-transcribe part of the target audios (for English, French and Spanish only) to allow more accurate alignments. To use them instead of machine transcriptions in the alignments, add --use-annotated-target to the command line.

Language Modeling (LM) Data

We combine VoxPopuli transcripts and text data from Europarl for LM training.

Download VoxPopuli and Europarl text data, process the raw text and generate the vocabulary via

python -m voxpopuli.get_lm_data --root [ROOT] --lang [LANGUAGE]

which outputs

  • sentences ${ROOT}/lm_data/[language]/sentences.txt
  • vocabulary ${ROOT}/lm_data/[language]/vocabulary.txt

To train an n-gram LM with KenLM, run

${KENLM_PATH}/lmplz -o ${n} --limit_vocab_file [OUT_VOCAB_FILE] < [OUT_TEXT_FILE] > ${n}gram_lm.arpa
${KENLM_PATH}/build_binary ${n}gram_lm.arpa ${n}gram_lm.bin

Pre-trained Models

wav2vec 2.0

We provide pre-trained wav2vec 2.0 models (implemented in fairseq and wav2letter/flashlight) for downstream speech tasks:

Language(s) Pre-training Hours Base Model (95M) Large Model (317M)
Es 4.4K fairseq fairseq
Fr 4.5K fairseq fairseq
It 4.6K fairseq fairseq
Nl 4.5K fairseq fairseq
Sv 4.5K fairseq fairseq
All 23 languages 10K fairseq fairseq
All 23 languages 100K fairseq / wav2letter fairseq

In our paper (Section 4.3.1), we evaluated these models on the Common Voice corpus in the normal setting and the few-shot phoneme recognition setting.

Wav2letter C++ implementation

A wav2letter implementation as well as a checkpoint pretrained on VoxPopuli 100k (base model) is also available in the Wav2letter respository.

The complete fine-tuned ASR baselines for this codebase shoulda come soon. The wav2letter implementation follows this paper.

ASR and LM

For the VoxPopuli ASR task, we provide Transformer baselines, fine-tuned wav2vec2 models (Base 10K) as well as n-gram LMs (trained with KenLM) and their lexicons:

Language ASR (fairseq) LM (kenLM) Lexicon
Cs baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
De baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
En baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Es baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Et baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Fi baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Fr baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Hr baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Hu baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
It baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Lt baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Nl baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Pl baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Ro baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Sk baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon
Sl baseline, fine-tuned wav2vec2 3-gram, 5-gram lexicon

We also provide CoVoST 2 + EuroParl-ST ASR Transformer models that are self-trained on 3000h VoxPopuli unlabelled data.

Language CoVoST 2 Test (WER) EuroParl-ST Test (WER) Model (fairseq)
De 17.3 21.4 s2t_transformer_l
Es 13.2 15.3 s2t_transformer_l
Fr 17.0 19.0 s2t_transformer_l

Please refer to the S2T examples for the use of Transformer model checkpoints.

Speech-to-Text Translation (ST)

We provide CoVoST 2 + EuroParl-ST ST Transformer models that are jointly trained with 400h VoxPopuli weakly labelled data.

Direction CoVoST 2 Test (BLEU) EuroParl-ST Test (BLEU) Model (fairseq)
De-En 23.4 24.4 s2t_transformer_l
Es-En 29.7 28.4 s2t_transformer_l
Fr-En 30.3 31.1 s2t_transformer_l

Please refer to the S2T examples for the use of these checkpoints.

What's New

  • 2021-07-26: New unlabelled data (additional 300K hours) released.
  • 2021-03-03: VoxPopuli released.

License

License
VoxPopuli Data CC0 (see also European Parliament's legal notice for the raw data)
LM Data (Please check out the Europarl website for the Europarl portion)
Pre-trained Models CC BY-NC 4.0
Code CC BY-NC 4.0

Contact

Changhan Wang (changhan@fb.com), Morgane Rivière (mriviere@fb.com), Ann Lee (annl@fb.com)

Citation

@inproceedings{wang-etal-2021-voxpopuli,
    title = "{V}ox{P}opuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation",
    author = "Wang, Changhan  and
      Riviere, Morgane  and
      Lee, Ann  and
      Wu, Anne  and
      Talnikar, Chaitanya  and
      Haziza, Daniel  and
      Williamson, Mary  and
      Pino, Juan  and
      Dupoux, Emmanuel",
    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.acl-long.80",
    pages = "993--1003",
}