yuriak/SpeechTransProgress

Tracking the progress in end-to-end speech translation

End-to-End Speech Translation Progress

Tutorial

• EACL 2021 tutorial: Speech Translation
• Blog: Getting Started with End-to-End Speech Translation
• ACL 2020 Theme paper: Speech Translation and the End-to-End Promise: Taking Stock of Where We Are
• INTERSPEECH 2019 survey talk: Spoken Language Translation

Data

Corpus	Direction	Target	Duration	License
CoVoST 2	{Fr, De, Es, Ca, It, Ru, Zh, Pt, Fa, Et, Mn, Nl, Tr, Ar, Sv, Lv, Sl, Ta, Ja, Id, Cy} -> En and En -> {De, Ca, Zh, Fa, Et, Mn, Tr, Ar, Sv, Lv, Sl, Ta, Ja, Id, Cy}	Text	2880h	CC0
mTEDx	{Es, Fr, Pt, It, Ru, El} -> En, {Fr, Pt, It} -> Es, Es -> {Fr, It}, {Es,Fr} -> Pt	Text	765h	CC BY-NC-ND 4.0
CoVoST	{Fr, De, Nl, Ru, Es, It, Tr, Fa, Sv, Mn, Zh} -> En	Text	700h	CC0
MUST-C & MUST-Cinema	En -> {De, Es, Fr, It, Nl, Pt, Ro, Ru, Ar, Cs, Fa, Tr, Vi, Zh}	Text	504h	CC BY-NC-ND 4.0
How2	En -> Pt	Text	300h	Youtube & CC BY-SA 4.0
Augmented LibriSpeech	En -> Fr	Text	236h	CC BY 4.0
Europarl-ST	{En, Fr, De, Es, It, Pt, Pl, Ro, Nl} -> {En, Fr, De, Es, It, Pt, Pl, Ro, Nl}	Text	280h	CC BY-NC 4.0
Kosp2e	Ko -> En	Text	198h	Mixed CC
Fisher + Callhome	Es -> En	Text	160h+20h	LDC
MaSS	{En, Es, Eu, Fi, Fr, Hu, Ro, Ru} -> {En, Es, Eu, Fi, Fr, Hu, Ro, Ru}	Text & Speech	172h	Bible.is
LibriVoxDeEn	De -> En	Text	110h	CC BY-NC-SA 4.0
BSTC	Zh -> En	Text	68h

Toolkit

• ESPNet-ST
• Fairseq S2T
• NeurST

Paper

2021

• [arXiv] Simultaneous Speech Translation for Live Subtitling: from Delay to Display
• [arXiv] Translatotron 2: Robust direct speech-to-speech translation
• [arXiv] Direct speech-to-speech translation with discrete units
• [arXiv] Efficient Transformer for Direct Speech Translation
• [arXiv] Zero-shot Speech Translation
• [arXiv] Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation
• [arXiv] UniST: Unified End-to-end Model for Streaming and Non-streaming Speech Translation
• [arXiv] Direct simultaneous speech to speech translation
• [ASRU] Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediates
• [ASRU] Assessing Evaluation Metrics for Speech-to-Speech Translation
• [INTERSPEECH] Impact of Encoding and Segmentation Strategies on End-to-End Simultaneous Speech Translation
• [EMNLP] Speechformer: Reducing Information Loss in Direct Speech Translation
• [EMNLP] Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translation
• [INTERSPEECH] End-to-end Speech Translation via Cross-modal Progressive Training
• [INTERSPEECH] CoVoST 2 and Massively Multilingual Speech-to-Text Translation
• [INTERSPEECH] The Multilingual TEDx Corpus for Speech Recognition and Translation
• [INTERSPEECH] Large-Scale Self-and Semi-Supervised Learning for Speech Translation
• [INTERSPEECH] Kosp2e: Korean Speech to English Translation Corpus
• [INTERSPEECH] AlloST: Low-resource Speech Translation without Source Transcription
• [INTERSPEECH] SpecRec: An Alternative Solution for Improving End-to-End Speech-to-Text Translation via Spectrogram Reconstruction
• [INTERSPEECH] Optimally Encoding Inductive Biases into the Transformer Improves End-to-End Speech Translation
• [INTERSPEECH] ASR Posterior-based Loss for Multi-task End-to-end Speech Translation
• [ACL] Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders
• [ACL] Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• [ACL] Lightweight Adapter Tuning for Multilingual Speech Translation
• [ACL] Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?
• [ACL] Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task
• [ACL Findings] Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR
• [ACL Findings] AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation
• [ACL Findings] RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer
• [ACL Findings] Learning Shared Semantic Space for Speech-to-Text Translation
• [ACL Findings] Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation
• [ACL Findings] How to Split: the Effect of Word Segmentation on Gender Bias in Speech Translation
• [ACL Demo] NeurST: Neural Speech Translation Toolkit
• [ICML] Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation
• [NAACL] Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation
• [NAACL] Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
• [NAACL AutoSimTrans] BSTC: A Large-Scale Chinese-English Speech Translation Dataset
• [AmericasNLP] Highland Puebla Nahuatl–Spanish Speech Translation Corpus for Endangered Language Documentation
• [ICASSP] Task Aware Multi-Task Learning for Speech to Text Tasks
• [ICASSP] A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
• [ICASSP] An Empirical Study of End-to-end Simultaneous Speech Translation Decoding Strategies
• [ICASSP] Streaming Simultaneous Speech Translation with Augmented Memory Transformer
• [ICASSP] Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
• [ICASSP] Cascaded Models With Cyclic Feedback For Direct Speech Translation
• [ICASSP] Jointly Trained Transformers models for Spoken Language Translation
• [ICASSP] Efficient Use of End-to-end Data in Spoken Language Processing
• [EACL] CTC-based Compression for Direct Speech Translation
• [EACL] Streaming Models for Joint Speech Recognition and Translation
• [IberSPEECH] mintzai-ST: Corpus and Baselines for Basque-Spanish Speech Translation
• [AAAI] Consecutive Decoding for Speech-to-text Translation
• [AAAI] UWSpeech: Speech to Speech Translation for Unwritten Languages
• [AAAI] "Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation
• [SLT] Tight Integrated End-to-End Training for Cascaded Speech Translation
• [SLT] Transformer-based Direct Speech-to-speech Translation with Transcoder

2020

• [arXiv] Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoders
• [arXiv] Bridging the Modality Gap for Speech-to-Text Translation
• [arXiv] CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning
• [CLiC-IT] On Knowledge Distillation for Direct Speech Translation
• [COLING] Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation
• [COLING] Breeding Gender-aware Direct Speech Translation Systems
• [AACL] SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation
• [AACL Demo] fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• [EMNLP] Effectively pretraining a speech translation decoder with Machine Translation data
• [EMNLP Findings] Adaptive Feature Selection for End-to-End Speech Translation
• [AMTA] On Target Segmentation for Direct Speech Translation
• [INTERSPEECH] Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection
• [INTERSPEECH] Relative Positional Encoding for Speech Recognition and Direct Translation
• [INTERSPEECH] Contextualized Translation of Automatically Segmented Speech
• [INTERSPEECH] Self-Training for End-to-End Speech Translation
• [INTERSPEECH] Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
• [INTERSPEECH] Self-Supervised Representations Improve End-to-End Speech Translation
• [INTERSPEECH] Investigating Self-Supervised Pre-Training for End-to-End Speech Translation
• [TACL] Consistent Transcription and Translation of Speech
• [ACL] Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation
• [ACL] Phone Features Improve Speech Translation
• [ACL] Curriculum Pre-training for End-to-End Speech Translation
• [ACL] SimulSpeech: End-to-End Simultaneous Speech to Text Translation
• [ACL] Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus
• [ACL Theme] Speech Translation and the End-to-End Promise: Taking Stock of Where We Are
• [ACL Demo] ESPnet-ST: All-in-One Speech Translation Toolkit
• [LREC] CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus
• [LREC] MuST-Cinema: a Speech-to-Subtitles corpus
• [LREC] MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Bible
• [LREC] LibriVoxDeEn: A Corpus for German-to-English Speech Translation and Speech Recognition
• [ICASSP] Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates
• [ICASSP] Instance-Based Model Adaptation For Direct Speech Translation
• [ICASSP] Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning
• [ICASSP] Analyzing ASR pretraining for low-resource speech-to-text translation
• [ICASSP] End-to-End Speech Translation with Self-Contained Vocabulary Manipulation
• [AAAI] Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation
• [AAAI] Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding

2019

• [ASRU] One-To-Many Multilingual End-to-end Speech Translation
• [ASRU] Multilingual End-to-End Speech Translation
• [ASRU] Speech-to-speech Translation between Untranscribed Unknown Languages
• [ASRU] A Comparative Study on End-to-end Speech to Text Translation
• [IWSLT] Harnessing Indirect Training Data for End-to-End Automatic Speech Translation: Tricks of the Trade
• [IWSLT] On Using SpecAugment for End-to-End Speech Translation
• [INTERSPEECH] End-to-End Speech Translation with Knowledge Distillation
• [INTERSPEECH] Adapting Transformer to End-to-end Spoken Language Translation
• [INTERSPEECH] Direct speech-to-speech translation with a sequence-to-sequence model
• [ACL] Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation
• [ACL] Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translation
• [NAACL] Pre-training on High-Resource Speech Recognition Improves Low-Resource Speech-to-Text Translation
• [NAACL] MuST-C: a Multilingual Speech Translation Corpus
• [NAACL] Fluent Translations from Disfluent Speech in End-to-End Speech Translation
• [ICASSP] Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
• [ICASSP] Towards unsupervised speech-to-text translation
• [ICASSP] Towards End-to-end Speech-to-text Translation with Two-pass Decoding

2018

• [NIPS] How2: A Large-scale Dataset for Multimodal Language Understanding
• [IberSPEECH] End-to-End Speech Translation with the Transformer
• [INTERSPEECH] Low-Resource Speech-to-Text Translation
• [LREC] Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation
• [NAACL] Tied multitask learning for neural speech translation
• [ICASSP] End-to-End Automatic Speech Translation of Audiobooks

2017

• [INTERSPEECH] Sequence-to-Sequence Models Can Directly Translate Foreign Speech
• [INTERSPEECH] Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translation
• [EACL] Towards speech-to-text translation without speech recognition

2016

• [NIPS Workshop] Listen and translate: A proof of concept for end-to-end speech-to-text translation
• [NAACL] An Attentional Model for Speech Translation Without Transcription

2013

• [IWSLT] Improved Speech-to-Text Translation with the Fisher and Callhome Spanish–English Speech Translation Corpus

Contact

Changhan Wang (wangchanghan@gmail.com)