/dla

Deep learning for audio processing

Primary LanguageJupyter NotebookMIT LicenseMIT

logo5v1

Deep Learning for Audio (DLA)

  • Lecture and seminar materials for each week are in ./week* folders, see README.md for materials and instructions
  • Any technical issues, ideas, bugs in course materials, contribution ideas - add an issue
  • The current version of the course is conducted in autumn 2023 at the CS Faculty of HSE

Syllabus

  • week01 Introduction to Course

    • Lecture: Introduction to Course
    • Seminar: Intro in pytorch

  • week02 Introduction to Digital Signal Processing

    • Lecture: Signals, Fourier Transform, spectrograms, MelScale, MFCC
    • Seminar: DSP in practice, spectrogram creation, training a model for audio MNIST

  • week03 Speech Recognition I

    • Lecture: Metrics, datasets, Connectionist Temporal Classification (CTC), Listen Attend and Spell (LAS), Beam Search
    • Seminar: Audio Augmentations, Beam Search, Homework discussion

  • week04 Speech Recognition II

    • Lecture: RNN-T, language model fusion, Byte-Pair Encoding (BPE)
    • Seminar: --

  • week05 Source Separation I

    • Lecture: A review of general Source Separation and Denoising, Encoder-Decoder-Separator architectures, Demucs family, DCCRN, FullSubNet+
    • Seminar: Metrics, Dataset of Mixtures and some tech stuff

  • week06 Source Separation II

    • Lecture: Speech separation, Blind and Target Separation, Recurrent(TasNet, DPRNN, VoiceFilter) and CNN(ConvTasNet, SpEx+)
    • Seminar: WienerFilter, SincFilter and DEMUCS

  • week07 Text to Speech (TTS)

    • Lecture: Tacotron, DeepVoice, GST, FastSpeech, AdaSpeech, Attention Tricks
    • Seminar: FastSpeech I

  • week08 Neural Vocoders

    • Lecture: WaveNet, Parallel WaveGAN, WaveGlow, MelGAN, HiFiGAN
    • Seminar: WaveNet

  • week09 Voice Conversion

    • Lecture: Disentanglement & Direct based methods
    • Seminar: TorchScript, HiFi-VC

  • week10 Voice Biometry I

    • Lecture: Introduction. CMs for sythesized speech detection (LCNN, RawNet2, AASIST). GNNs
    • Seminar: ASVspoof, Sinc-layer, GNN

  • week11 Voice Biometry II

    • Lecture: CMs for replay attack detection. ASV systems. SASV systems. Streaming
    • Seminar: -

  • week12 Diffusion Models for Audio Generation

    • Lecture, part 1: Introduction to diffusion models from two perspectives: score matching and latent probabilistic models.
    • Lecture, part2: Diffusion models for audio synthesis and tts. WaveGrad, DiffWave, GradTTS

  • bonus week Guest lecture

    • Self-Supervised models in ASR

Homeworks

  • ASR Training speech recognition model
  • SS Training speech separation model
  • TTS Implementation of TTS model (Part 1, FastSpeech)
  • NV Implementation of TTS model (Part 2, Vocoder)
  • AS Implementation of Anti-spoofing Model

Resources

Contributors & course staff

Course materials and teaching (in different years) were delivered by: