/speechbrain

A PyTorch-based Speech Toolkit

Primary LanguagePythonApache License 2.0Apache-2.0

SpeechBrain Logo

Typing SVG

| ๐Ÿ“˜ Tutorials | ๐ŸŒ Website | ๐Ÿ“š Documentation | ๐Ÿค Contributing | ๐Ÿค— HuggingFace | โ–ถ๏ธ YouTube | ๐Ÿฆ X |

GitHub Repo stars Please, help our community project. Star on GitHub!

Exciting News (January, 2024): Discover what is new in SpeechBrain 1.0 here!

๐Ÿ—ฃ๏ธ๐Ÿ’ฌ What SpeechBrain Offers

  • SpeechBrain is an open-source PyTorch toolkit that accelerates Conversational AI development, i.e., the technology behind speech assistants, chatbots, and large language models.

  • It is crafted for fast and easy creation of advanced technologies for Speech and Text Processing.

๐ŸŒ Vision

  • With the rise of deep learning, once-distant domains like speech processing and NLP are now very close. A well-designed neural network and large datasets are all you need.

  • We think it is now time for a holistic toolkit that, mimicking the human brain, jointly supports diverse technologies for complex Conversational AI systems.

  • This spans speech recognition, speaker recognition, speech enhancement, speech separation, language modeling, dialogue, and beyond.

๐Ÿ“š Training Recipes

  • We share over 200 competitive training recipes on more than 40 datasets supporting 20 speech and text processing tasks (see below).

  • We support both training from scratch and fine-tuning pretrained models such as Whisper, Wav2Vec2, WavLM, Hubert, GPT2, Llama2, and beyond. The models on HuggingFace can be easily plugged in and fine-tuned.

  • For any task, you train the model using these commands:

python train.py hparams/train.yaml
  • The hyperparameters are encapsulated in a YAML file, while the training process is orchestrated through a Python script.

  • We maintained a consistent code structure across different tasks.

  • For better replicability, training logs and checkpoints are hosted on Dropbox.

drawing Pretrained Models and Inference

  • Access over 100 pretrained models hosted on HuggingFace.
  • Each model comes with a user-friendly interface for seamless inference. For example, transcribing speech using a pretrained model requires just three lines of code:
from speechbrain.inference import EncoderDecoderASR

asr_model = EncoderDecoderASR.from_hparams(source="speechbrain/asr-conformer-transformerlm-librispeech", savedir="pretrained_models/asr-transformer-transformerlm-librispeech")
asr_model.transcribe_file("speechbrain/asr-conformer-transformerlm-librispeech/example.wav")

drawing Documentation

  • We are deeply dedicated to promoting inclusivity and education.
  • We have authored over 30 tutorials on Google Colab that not only describe how SpeechBrain works but also help users familiarize themselves with Conversational AI.
  • Every class or function has clear explanations and examples that you can run. Check out the documentation for more details ๐Ÿ“š.

๐ŸŽฏ Use Cases

  • ๐Ÿš€ Research Acceleration: Speeding up academic and industrial research. You can develop and integrate new models effortlessly, comparing their performance against our baselines.

  • โšก๏ธ Rapid Prototyping: Ideal for quick prototyping in time-sensitive projects.

  • ๐ŸŽ“ Educational Tool: SpeechBrain's simplicity makes it a valuable educational resource. It is used by institutions like Mila, Concordia University, Avignon University, and many others for student training.

๐Ÿš€ Quick Start

To get started with SpeechBrain, follow these simple steps:

๐Ÿ› ๏ธ Installation

Install via PyPI

  1. Install SpeechBrain using PyPI:

    pip install speechbrain
  2. Access SpeechBrain in your Python code:

    import speechbrain as sb

Install from GitHub

This installation is recommended for users who wish to conduct experiments and customize the toolkit according to their needs.

  1. Clone the GitHub repository and install the requirements:

    git clone https://github.com/speechbrain/speechbrain.git
    cd speechbrain
    pip install -r requirements.txt
    pip install --editable .
  2. Access SpeechBrain in your Python code:

    import speechbrain as sb

Any modifications made to the speechbrain package will be automatically reflected, thanks to the --editable flag.

โœ”๏ธ Test Installation

Ensure your installation is correct by running the following commands:

pytest tests
pytest --doctest-modules speechbrain

๐Ÿƒโ€โ™‚๏ธ Running an Experiment

In SpeechBrain, you can train a model for any task using the following steps:

cd recipes/<dataset>/<task>/
python experiment.py params.yaml

The results will be saved in the output_folder specified in the YAML file.

๐Ÿ“˜ Learning SpeechBrain

  • Website: Explore general information on the official website.

  • Tutorials: Start with basic tutorials covering fundamental functionalities. Find advanced tutorials and topics in the Tutorials menu on the SpeechBrain website.

  • Documentation: Detailed information on the SpeechBrain API, contribution guidelines, and code is available in the documentation.

๐Ÿ”ง Supported Technologies

  • SpeechBrain is a versatile framework designed for implementing a wide range of technologies within the field of Conversational AI.
  • It excels not only in individual task implementations but also in combining various technologies into complex pipelines.

๐ŸŽ™๏ธ Speech/Audio Processing

Tasks Datasets Technologies/Models
Speech Recognition AISHELL-1, CommonVoice, DVoice, KsponSpeech, LibriSpeech, MEDIA, RescueSpeech, Switchboard, TIMIT, Tedlium2, Voicebank CTC, Transducers, Transformers, Seq2Seq, Beamsearch techniques for CTC,seq2seq,transducers), Rescoring, Conformer, Branchformer, Hyperconformer, Kaldi2-FST
Speaker Recognition VoxCeleb ECAPA-TDNN, ResNET, Xvectors, PLDA, Score Normalization
Speech Separation WSJ0Mix, LibriMix, WHAM!, WHAMR!, Aishell1Mix, BinauralWSJ0Mix SepFormer, RESepFormer, SkiM, DualPath RNN, ConvTasNET
Speech Enhancement DNS, Voicebank SepFormer, MetricGAN, MetricGAN-U, SEGAN, spectral masking, time masking
Text-to-Speech LJSpeech, LibriTTS Tacotron2, Zero-Shot Multi-Speaker Tacotron2, FastSpeech2
Vocoding LJSpeech, LibriTTS HiFiGAN, DiffWave
Spoken Language Understanding MEDIA, SLURP, Fluent Speech Commands, Timers-and-Such Direct SLU, Decoupled SLU, Multistage SLU
Speech-to-Speech Translation CVSS Discrete Hubert, HiFiGAN, wav2vec2
Speech Translation Fisher CallHome (Spanish), IWSLT22(lowresource) wav2vec2
Emotion Classification IEMOCAP, ZaionEmotionDataset ECAPA-TDNN, wav2vec2, Emotion Diarization
Language Identification VoxLingua107, CommonLanguage ECAPA-TDNN
Voice Activity Detection LibriParty CRDNN
Sound Classification ESC50, UrbanSound CNN14, ECAPA-TDNN
Self-Supervised Learning CommonVoice, LibriSpeech wav2vec2
Interpretability ESC50 Learning-to-Interpret (L2I), Non-Negative Matrix Factorization (NMF), PIQ
Speech Generation AudioMNIST Diffusion, Latent Diffusion
Metric Learning REAL-M, Voicebank Blind SNR-Estimation, PESQ Learning
Alignment TIMIT CTC, Viterbi, Forward Forward
Diarization AMI ECAPA-TDNN, X-vectors, Spectral Clustering

๐Ÿ“ Text Processing

Tasks Datasets Technologies/Models
Language Modeling CommonVoice, LibriSpeech n-grams, RNNLM, TransformerLM
Response Generation MultiWOZ GPT2, Llama2
Grapheme-to-Phoneme LibriSpeech RNN, Transformer, Curriculum Learning, Homograph loss

๐Ÿ” Additional Features

SpeechBrain includes a range of native functionalities that enhance the development of Conversational AI technologies. Here are some examples:

  • Training Orchestration: The Brain class serves as a fully customizable tool for managing training and evaluation loops over data. It simplifies training loops while providing the flexibility to override any part of the process.

  • Hyperparameter Management: A YAML-based hyperparameter file specifies all hyperparameters, from individual numbers (e.g., learning rate) to complete objects (e.g., custom models). This elegant solution drastically simplifies the training script.

  • Dynamic Dataloader: Enables flexible and efficient data reading.

  • GPU Training: Supports single and multi-GPU training, including distributed training.

  • Dynamic Batching: On-the-fly dynamic batching enhances the efficient processing of variable-length signals.

  • Mixed-Precision Training: Accelerates training through mixed-precision techniques.

  • Efficient Data Reading: Reads large datasets efficiently from a shared Network File System (NFS) via WebDataset.

  • Hugging Face Integration: Interfaces seamlessly with HuggingFace for popular models such as wav2vec2 and Hubert.

  • Orion Integration: Interfaces with Orion for hyperparameter tuning.

  • Speech Augmentation Techniques: Includes SpecAugment, Noise, Reverberation, and more.

  • Data Preparation Scripts: Includes scripts for preparing data for supported datasets.

SpeechBrain is rapidly evolving, with ongoing efforts to support a growing array of technologies in the future.

๐Ÿ“Š Performance

  • SpeechBrain integrates a variety of technologies, including those that achieves competitive or state-of-the-art performance.

  • For a comprehensive overview of the achieved performance across different tasks, datasets, and technologies, please visit here.

๐Ÿ“œ License

  • SpeechBrain is released under the Apache License, version 2.0, a popular BSD-like license.
  • You are free to redistribute SpeechBrain for both free and commercial purposes, with the condition of retaining license headers. Unlike the GPL, the Apache License is not viral, meaning you are not obligated to release modifications to the source code.

๐Ÿ”ฎFuture Plans

We have ambitious plans for the future, with a focus on the following priorities:

  • Scale Up: Our aim is to provide comprehensive recipes and technologies for training massive models on extensive datasets.

  • Scale Down: While scaling up delivers unprecedented performance, we recognize the challenges of deploying large models in production scenarios. We are focusing on real-time, streamable, and small-footprint Conversational AI.

๐Ÿค Contributing

  • SpeechBrain is a community-driven project, led by a core team with the support of numerous international collaborators.
  • We welcome contributions and ideas from the community. For more information, check here.

๐Ÿ™ Sponsors

  • SpeechBrain is an academically driven project and relies on the passion and enthusiasm of its contributors.
  • As we cannot rely on the resources of a large company, we deeply appreciate any form of support, including donations or collaboration with the core team.
  • If you're interested in sponsoring SpeechBrain, please reach out to us at speechbrainproject@gmail.com.
  • A heartfelt thank you to all our sponsors, including the current ones:

Image 1     Image 3     Image 4



Image 5     Image 2     Image 6



Image 7     Image 9     Image 8    

๐Ÿ“– Citing SpeechBrain

If you use SpeechBrain in your research or business, please cite it using the following BibTeX entry:

@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and Franรงois Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}