megazone87/ctc-benchmark

CTC Benchmark

---

What's the best CTC implementation?

About

This project aims to benchmark the performance of some ctc algorithm implementations.

Currently include:

• torch.nn.CTCLoss
• tf.nn.ctc_loss
• k2.CtcLoss

Getting Started

pip install -r requirements.txt

Running the benchmark tests

# run ctc:forward (get ctc-loss) tests
pytest -k "forward" --benchmark-columns='mean' .
# run ctc:backward (calculate gradients) tests
pytest -k "backward" --benchmark-columns='mean' .

Results

• ctc forward benchmark results

• ctc backward benchmark results

Versions

• TF: v2.6.0
• torch: v1.8.2
• k2: v1.9

Explainations

• TF test in eager execution mode, this may introduce the overhead
• K2 test at python API level as others, which include py->c++ call cost. K2 uses pybind11 as the binding framework, which may introduce the overhead.
• CTC implementation of CUDNN is the backend of torch_with_cudnn and tf_with_gpu

Todo

• Remove TF eager mode overhead from benchmark results.
• Remove K2 pybind11 overhead between python API -> c++ API from benchmark results, through call c++ API directly.
• Add GTN ctc alternative.
• Add comparations about more aspects of those CTC implementations.

Built Using

• Pytest - Python Test Framework
• Pytest-benchmark - Benchmark Framework within Pytest