This application is an implement of off-line spoken keyword spotting using MLI Lib based on ARC IoT Development Kit.The optimized convolutional neural network can figure out ten basic control commands,even though the processing unit is not that powerful.Then we can use voice to control many things,for example,a snake game.
DEMO VIDEO:Video presentation
- Spotting the ten spoken commands:yes/no/up/down/left/right/on/off/stop/go.And the word will be printed through uart.
- Play the game!Saying up/down/left/right commands,you can change the direction of the snake to catch the food or avoid crash with itself.
- ARC GNU Toolset 2018.09
- emcARC OSP 2018.12 with MLI Lib
- Serial port terminal, such as putty, tera-term or minicom
1.MAX7219 pins.
MAX7219 pins | IoT DK pins |
---|---|
CS | GPIO_8b0_PIN4 |
DATA | GPIO_8b0_PIN5 |
CLK | GPIO_8b1_PIN4 |
2.Connect I2C pins of WM8978 to I2C0 of IoT DK.
3.Connect I2S pins of WM8978 to I2S_RX of IoT DK.The REF_CLK should not be connected untile the initialization of WM8978 is finished.
- Download source code of SoundSnake from github.
- Make sure all the connections are correct.
- The link script file has been modified to ensure the heap and stack are large enough and the addresses of program sections have also been redirected.Make sure the link script is correct.
- Open serial terminal and configure it to right COM port and 115200bps.
- Download with USB-JTAG or use bootloader to boot the program.
- Connect the MCLK(REF_CLK) wire when seeing "Please connet the mclk !"at serial terminal.
- Speak to the Mic.You'll see the word at the serial terminal and the snake will follow your command to change direction.
-
Selected MLI Lib here, then you can use MLI Lib API in your application:
LIB_SEL = embarc_mli
-
Target options about IoT DK and toolchain:
BOARD = iotdk BD_VER = 10 TOOLCHAIN = gnu
-
The middleware used in your application:
MID_SEL = common
See embARC Example User Guide, "Options to Hard-Code in the Application Makefile" for more detailed information about Makefile Options.
file | Function |
---|---|
main.c | Initialization and main loop |
kws.c/kws.h | Keyword spotting(CNN) |
ds_cnn_weights.h | Weights list of CNN |
mfcc.c/mfcc.h | Algorithm of extracting mfcc features |
codec.c/codec.h | Recive and store audio frames(interrupt) |
FFT.c/FFT.h | Fast Fourier transform |
wm8978.c/wm8978.h | Config wm8978 through I2C |
wm8978i2s.c/.h | I2S driver |