TensorflowASR

State-of-the-art Automatic Speech Recognition in Tensorflow 2

CTC\Transducer\LAS Default is Chinese ASR

Now the project is still in the development stages

Welcome to use and feedback bugs

What's New?

New:

Add Mel Layer support training

All Structure add mel layerIt's more like end-to-end,now you can feed wav to model

am_data.yml

use_mel_layer: True
mel_layer_type: Melspectrogram #Melspectrogram
trainable_kernel: True #support train model

Last:

Fix LAS stop_loss bug
Change CTC tf.nn.ctc_loss to tf.keras.backend.ctc_batch_cost
- found tf.nn.ctc_loss can't play the right role
- So the blank must be n_classes+1,blank_at_zero set False

Future

pre-train model
Fix bugs

Supported Structure

CTC
Transducer
LAS
MultiTaskCTC

Supported Models

Conformer
ESPNet:Efficient Spatial Pyramid of Dilated Convolutions
DeepSpeech2
Transformer Pinyin to Chinese characters
- O2O-Encoder-Decoder Complete transformer,and one to one relationship between phoneme and target ,e.g.: pin4 yin4-> 拼音
- O2O-Encoder Not contain the decoder part,others are same.
- Encoder-Decoder Typic transformer

Requirements

Python 3.6+
Tensorflow 2.2+: pip install tensorflow
librosa
pypinyin if you need use the default phoneme
keras-bert
addons For LAS structure,pip install tensorflow-addons
tqdm
jieba
wrap_rnnt_loss not essential,provide in ./externals
wrap_ctc_decoders not essential,provide in ./externals

Usage

Prepare train_list.

am_train_list format:

file_path1 \t text1
file_path2 \t text2
……

lm_train_list format:

text1
text2
……

Down the bert model for LM training,if you don't need LM can skip this Step:
```
 https://pan.baidu.com/s/1_HDAhfGZfNhXS-cYoLQucA extraction code: 4hsa
```
Modify the am_data.yml (in ./configs),set running params.Modify the name in model yaml to choose the structure.

Just run:

python train_am.py --data_config ./configs/am_data.yml --model_config ./configs/conformer.yml

To Test,you can follow in run-test.py,addition,you can modify the predict function to meet your needs:

from utils.user_config import UserConfig
from AMmodel.model import AM
from LMmodel.trm_lm import LM

am_config=UserConfig(r'./configs/am_data.yml',r'./configs/conformer.yml')
lm_config = UserConfig(r'./configs/lm_data.yml', r'./configs/transformer.yml')

am=AM(am_config)
am.load_model(training=False)

lm=LM(lm_config)
lm.load_model()

am_result=am.predict(wav_path)

lm_result=lm.predict(am_result)

Use Tester to test your model: Fisrt modify the eval_list in am_data.yml/lm_data.yml

Then:

python eval_am.py --data_config ./configs/am_data.yml --model_config ./configs/conformer.yml

Tester will show SER/CER/DEL/INS/SUB

Your Model

You can add your model in ./AMmodel folder e.g, LM model is the same with follow:

from AMmodel.transducer_wrap import Transducer
from AMmodel.ctc_wrap import CtcModel
from AMmodel.las_wrap import LAS,LASConfig
class YourModel(tf.keras.Model):
    def __init__(self,……):
        super(YourModel, self).__init__(……)
        ……
    
    def call(self, inputs, training=False, **kwargs):
       
        ……
        return decoded_feature
        
#To CTC
class YourModelCTC(CtcModel):
    def __init__(self,
                ……
                 **kwargs):
        super(YourModelCTC, self).__init__(
        encoder=YourModel(……),num_classes=vocabulary_size,name=name,
        )
        self.time_reduction_factor = reduction_factor #if you never use the downsample layer,set 1

#To Transducer
class YourModelTransducer(Transducer):
    def __init__(self,
                ……
                 **kwargs):
        super(YourModelTransducer, self).__init__(
            encoder=YourModel(……),
            vocabulary_size=vocabulary_size,
            embed_dim=embed_dim,
            embed_dropout=embed_dropout,
            num_lstms=num_lstms,
            lstm_units=lstm_units,
            joint_dim=joint_dim,
            name=name, **kwargs
        )
        self.time_reduction_factor = reduction_factor #if you never use the downsample layer,set 1

#To LAS
class YourModelLAS(LAS):
    def __init__(self,
                ……,
                config,# the config dict in model yml
                training,
                 **kwargs):
        config['LAS_decoder'].update({'encoder_dim':encoder_dim})# encoder_dim is your encoder's last dimension
        decoder_config=LASConfig(**config['LAS_decoder'])

        super(YourModelLAS, self).__init__(
        encoder=YourModel(……),
        config=decoder_config,
        training=training,
        )
        self.time_reduction_factor = reduction_factor #if you never use the downsample layer,set 1

Then,import the your model in ./AMmodel/model.py ,modify the load_model function

Convert to pb

AM/LM model are the same as follow:

from AMmodel.model import AM
am_config = UserConfig('...','...')
am=AM(am_config)
am.load_model(False)
am.convert_to_pb(export_path)

Tips

IF you want to use your own phoneme,modify the convert function in am_dataloader.py/lm_dataloader.py

def init_text_to_vocab(self):#keep the name
    
    def text_to_vocab_func(txt):
        return your_convert_function

    self.text_to_vocab = text_to_vocab_func #here self.text_to_vocab is a function,not a call

Don't forget that the token list start with S and /S,e.g:

    S
    /S
    de
    shì
    ……

Performerce

The test data are aishell's test dataset and dev dataset.

Am takes the Pinyin phoneme as the final result and use CER (character error rate) to test.

LM is based on Chinese characters ,and use CER too.

After 10 epochs:

AM:

Test	Dev
4.1%	3.26%

LM:

Test	Dev
3.12%	3.16%

AM-LM:

Test	Dev
8.42%	7.36%

AM Speed Test,use a ~4.1 seconds wav on CPU:

CTC	Transducer	LAS
150ms	350ms	280ms

LM Speed Test,12 word on CPU:

O2O-Encoder-Decoder	O2O-Encoder	Encoder-Decoder
100ms	20ms	300ms

sirilanka/TensorflowASR