An audio synthesis module for prosody generation. It takes emotion and melody as inputs and maps the emotion to multiple timbre parameters (e.g. formants, attack time, spectrum features) which then be sent to the software synthesizer implemented on SuperCollider.
Besides the emotion input, the module also supports independent microtone, vibrato, brightness and noisiness controls.
The following audio clips are generated by the system. (The melodies are pre-defined.)
This is the screen shot of the control panel made in SuperCollider:
The user can select an emotion using the 2d slider on the left. The emotion is modeled by 2 dimensions: valence and power. Check out Geneva Emotion Wheel to specify a specific emotion. The other parameters can be controlled independently.
These are the two main references of this project:
Formant modulation:
- Goudbeek, M., Goldman, J. P., & Scherer, K. R. (2009). Emotion dimensions and formant position. In Tenth Annual Conference of the International Speech Communication Association.
Timbre modulation (attack, hf-lf ratio)
- Eerola, T., Ferrer, R., & Alluri, V. (2012). Timbre and affect dimensions: Evidence from affect and similarity ratings and acoustic correlates of isolated instrument sounds. Music Perception: An Interdisciplinary Journal, 30(1), 49-70
This project provides:
- A SuperCollider
.scd
file that comes with aSynthDef
and OSC receiving functionality. - A SuperCollider GUI control panel.
- Two Python OSC modules that handle GUI-to-Synth and MIDI-to-Synth respectively.
- A Python prosody feature mapping module.
Workflow:
SuperCollider GUI -> (OSC) -> Feature Mapping and OSC Transmit -> (OSC) -> SuperCollider Synth
or
MIDI -> (mido) -> Feature Mapping and OSC Transmit -> (OSC) -> SuperCollider Synth
Make sure you have Python and SuperCollider installed.
In python/
directory, run the following command to install Python dependencies:
$ pip install -r requirements.txt
Most of the time, you need a configuration file to make it work properly. To do
this, put a file config.yml
in the project root. Here's a configuration
example:
osc_bridge_ip: 127.0.0.1
osc_bridge_port: 8088
sc_ip: 127.0.0.1
sc_port: 57120
midi_device: MPKmini2
Usually you only need to change the sc_port
and midi_device
entries.
To check all the available devices on your computer, go to your Python REPL, type in the following commands:
>>> import mido
>>> mido.get_output_names()
Run sclang/vsynth_gui.scd
on SuperCollider. You can run this either on
SuperCollider IDE or through command line. For command line, make sure you are
at the project root directory, and use the following command:
$ /path/to/sclang sclang/vsynth_gui.scd
Also run python/gui_to_sclang
to enable the OSC transmit.
Make sure you have installed all the required packages. Run
python/midi_to_sclang
to enable MIDI input. If there's no more argument, the
program reads only MIDI messages from the hardware. You can also give it a file
path, such as ../midi
, where the program will watch the file change and send
the MIDI messages stored in the newly-created MIDI file.
Some CC numbers are used to send control parameters:
- CC1 Valence
- CC2 Power
- CC3 Tune (microtone control, ±100 cents, value 64 corresponds to 0 cent)
- CC4 Vibrato
- CC5 Brightness
- CC6 Noisiness
Run sclang/vsynth_osc.scd
on SuperCollider. You can run this either on
SuperCollider IDE or through command line. For command line, make sure you are
at the project root directory, and use the following command:
$ /path/to/sclang sclang/vsynth_osc.scd
Once you start running the code, there's one line in the output that tells you
the port SuperCollider is listening on (Listening on: xxxxx
). Then check your
config.yml
, make sure the port specified in sc_port
entry is the same as
this port.