/dla

Deep learning for audio processing

Primary LanguageJupyter NotebookMIT LicenseMIT

Deep Learning for Audio (DLA)

  • Lecture and seminar materials for each week are in ./week* folders, see README.md for materials and instructions
  • Any technical issues, ideas, bugs in course materials, contribution ideas - add an issue
  • The current version of the course is conducted in autumn 2020 at the CS Faculty of HSE

Syllabus

  • week01 Introduction to Digital Signal Processing

    • Lecture: Signals, Fourier transform, Spectrograms, MFCC and etc
    • Seminar: Intro in PyTorch, DevOps, R&D in Deep Learning

  • week02 Automatic Speech Recognition I

    • Lecture: Metrics, Attention, LAS, CTC, BeamSearch
    • Seminar: Docker, W&B, Augmentations for Audio

  • week03 Automatic Speech Recognition II

    • Lecture: LM Fusing, RNN Transducer, Schedule Sampling, BPE
    • Seminar: Jasper, QurtzNet, Mixed Precision Training, DDP/DP

  • week04 Key-word spottind (KWS) and Voice Activity Detection (VAD)

    • Lecture: (DNN, CNN, RNN+Attention) based KWS, SVDF, Orthogonality Regularization and other Tricks
    • Seminar: Speeding Up NNs: Tensor Decomposition, Quantization, Pruning, Distilation and Architecture Design

  • week05 Speaker verification and identification

    • Lecture: Metric Learning: Cosine, Contrastive, Triplet Losses. Angular Softmax. ArcFace
    • Seminar: Generalized End2End Loss for Speaker Verification

  • week06 Text to Speech

    • Lecture: Tacotron, DeepVoice, GST, FastSpeech, Attention Tricks
    • Seminar: Location-Sensitive Attention

  • week07 Neural Vocoders

    • Lecture: Introduction into generative models: AR, GAN, NF. WaveNet, ParallelWaveNet, WaveGlow, WaveFlow, MelGAN, PWG.

  • week08 Voice Conversion

    • Lecture: AutoVC, ConVoice, TTS Skins, StarGAN-VC-1-2, CycleGAN-1-2-3, Blow

  • week09 Music Generation

    • Lecture: VQVAE, Sparse Transformer, MuseNet, JukeBox

  • week10 Speech Enhancement, Denoising and Speaker Diarization

    • Lecture: SEGAN, TF Masking, HiFi Denoising, Speaker Diarization, VAD

  • week11 Self-supervision in Audio and Speech

    • Lecture: Intro to SS Learning. InfoNCE, CPC

Homeworks

  • DSP Implementation of basic ops like FFT, Spectrogram and MelScale

  • ASR Implementation of small ASR model, beam search and LM fusing

  • KWS Implementation of attention based KWS model, streaming scoring and model distillation

  • TTS Implementation of TTS model with different tricks

  • NV Implementation of Neural Vocoder Model

Contributors & course staff

Course materials and teaching performed by