Sound generated by facial expressions In this project, I wanted to detect human's face and make some sounds due to the emotions, I used clmtrackr to get the value of six emotions, happy, angry, disgused, surprised, fear and sad. Sometimes the emotion predictions were not accurate, but by multiplying them and calculate them to the pitches of notes, it could control sounds. Also, I added some beats effect and extra notes than the main loop recording to major or minor in music to enrich the melody. To generate sounds, I mainly used sampler and transport functions in Tone.js.
You could watch the remix video through this link: https://editor.p5js.org/dy2046/sketches/mZ0AD0v5W
P5 Editor(Syn Version): https://editor.p5js.org/dy2046/sketches/mZ0AD0v5W
P5 present: https://editor.p5js.org/dy2046/present/mZ0AD0v5W
Play through this: https://candice-lt.github.io/emotion_sound/
During the process, I changed poseNet.js to handpose.js to realize detection of hands’ position, and found index fingertop to trigger the notes to make sound. Also, I used tone.js instead of sound.js, cause Tone.js has sample and transport.repeat functions which fit my final projects’ sketch well.
Also, I struggled a lot with Audio Context, and finally fixed it by start Tone. after mouse is clicked and toggle it every time the mouse is clicked.
Referred to: https://tonejs.github.io/docs/r11/Transport
I used the function below to repeatly detect which note to produce sound.
Tone.Transport.scheduleRepeat(callback, "8n", "1m");
Next steps:
- to add more visual effects
- add more synthsized sounds