This is an implementation of the paper "Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention" https://arxiv.org/abs/1710.08969
To understand this notebook properly, you must have some knowledge about how sounds are dealt with, in data science and the various operations that are performed to convert a Waveform to a representaion suitable for analysis. Some Basic Concepts Are Mentioned Below :-
Before we start with what Fourier Trasnform is, let us first understand some terminologies that we will need to know about to understand it in a better way
A WaveForm or an audio signal is the raw representation of a sound wave as a function of Time and it's Intensities. This representaion is also known as the Time-Domain Representation or the Time-Amplitude Representation
A Frequency-Domain Representation refers to the analysis of mathematical functions or signals with respect to frequency, rather than time. This representaion is also known as the Frequency-Amplitude Representation or the Amplitude-Frequency Representation.
Put simply, a time-domain graph shows how a signal changes over time, whereas a frequency-domain graph shows how much of the signal lies within each given frequency band over a range of frequencies.
The Fourier transform (FT) decomposes a function of time (a signal) into its constituent frequencies. This is similar to the way a musical chord can be expressed in terms of the volumes and frequencies of its constituent notes. The term Fourier transform refers to both the frequency domain representation and the mathematical operation that associates the frequency domain representation to a function of time. Essentially A Fourier Trasnform when applied on a Signal Represented in the Time-Domain gets converted to the Frequency-Domain.
Now the question arieses, why do we need to convert the signal to a Frequency-Domain representation ?
Time domain signals, just provide the information regarding the value of a function/signal at any given instance. They do not convey the information as to the rate at which the signal is varying. Thus the need arises to represent the signal in another domain, describing the rate at which they vary, or the frequency.
But we need both the aspects of Time-Domain as well as the Frequency-Domain to understand better the properties the WaveForm that we have. The Time-Domain Representation gives us the times at which the signal was perceived while the Frequency-Domain Representation tells us about the Frequency distribution of the WaveForm.
The short-time Fourier transform (STFT), is a Fourier-related transform used to determine the sinusoidal frequency and phase content of local sections of a signal as it changes over time. In practice, the procedure for computing STFTs is to divide a longer time signal into shorter segments of equal length and then compute the Fourier transform separately on each shorter segment. This reveals the Fourier spectrum on each shorter segment. One then usually plots the changing spectra as a function of time.
The STFT is one of the most frequently used tools in speech analysis and processing. It describes the evolution of frequency components over time. Like the spectrum itself, one of the benefits of STFTs is that its parameters have a physical and intuitive interpretation.
A further parallel with a spectrum is that the output of the STFT is complex-valued, though where the spectrum is a vector, the STFT output is a matrix. As a consequence, we cannot directly visualize the complex-valued output. Instead, STFTs are usually visualized using their log-spectra,
Let’s forget for a moment about all these lovely visualization and talk math. The Mel Scale, mathematically speaking, is the result of some non-linear transformation of the frequency scale. This Mel Scale is constructed such that sounds of equal distance from each other on the Mel Scale, also “sound” to humans as they are equal in distance from one another. In contrast to Hz scale, where the difference between 500 and 1000 Hz is obvious, whereas the difference between 7500 and 8000 Hz is barely noticeable.
We know now what is a Spectrogram, and also what is the Mel Scale, so the Mel Spectrogram, is, rather surprisingly, a Spectrogram with the Mel Scale as its y axis.
LJSpeech dataset : Here The dataset will be downloaded when you run the notebook
Pre-Trained models : Here
The DC-TTS Model consits of 2 main Model:
We train this model to synthesize a coarse MEL Spectogram from a given Text.
This model consists of
Encodes the input sentence
Encodes the coarse MEL-Spectogram of previously spoken speech, whose length is
An Attention Matrix
$A = softmax($KT
Ant ~ 1 implies that the modules is looking at nth Character cn at time frame t and it will look at cn or cn+1 characters around them, at the subsequent time frame t+1.
The resultant
The result
Super resolution is the process of upscaling and or improving the details within an image. Often a low resolution image is taken as an input and the same image is upscaled to a higher resolution, which is the output. The details in the high resolution output are filled in where the details are essentially unknown.
Super resolution is essentially what you see in films and series like CSI where someone zooms into an image and it improves in quality and the details just appear.
Similarly our SSR model does the same but for audio signals or spectograms. This model converts the coarse MEL-Spectogram generated by the Text2Mel Model to A Super-Resolution Spectogram.
