Picovoice/speech-to-intent-benchmark

benchmark for Speech-to-Intent engines

Speech-to-Intent Benchmark

Made in Vancouver, Canada by Picovoice

This framework benchmarks the accuracy of Picovoice's Speech-to-Intent engine, Rhino. It compares the accuracy of Rhino with:

• Amazon Lex
• Google Dialogflow
• IBM Watson
• Microsoft LUIS

Table of Contents

• Speech-to-Intent Benchmark
  • Table of Contents
  • Results
  • Data
  • How to Reproduce?

Results

Command acceptance rate is the probability of an engine correctly understanding the spoken command. Below is the summary:

The figure below depicts engines performance at each SNR:

Data

The speech data are crowd-sourced from more than 50 unique speakers. Each speaker contributed about ten different utterances. Collectively there are 619 commands used in this benchmark. We test the engines in noisy conditions to simulate real-world situations. Noise is from Freesound.

How to Reproduce?

Clone the repository:

git clone https://github.com/Picovoice/speech-to-intent-benchmark.git

Get the usage message:

python3 src/bench.py --help

Then run the script for each engine.