Rhubarb Lip Sync T.P.I. for Godot is a Godot Engine addon made by AniMesuro that integrates Rhubarb Lip Sync's command-line program into a more user friendly popup-window optimized for use in Godot.
Rhubarb Lip Sync is a tool created by Daniel Wolf that allows for an automated alternative to producing lip sync animations. You can use it for animating speech in computer games, animated cartoons, or any similar project.
You can use the user interface window by browsing in Godot [Project > Tools > Rhubarb Lipsync TPI] or using the functions directly by calling the plugin instance. (see below)
- Table of Contents
- Usage
- Speech Recognizer
- Mouth Libraries
- Cleaning Routine
- File Checking Timer
- Screenshots
- Videos
- FileSelectorPreview
- Audio Slicing
- Calling functions by Code.
- Function Usage
- Disclaimer
- Bug Report
- When you first install Rhubarb Lip Sync TPI for Godot, you can use it by browsing in Godot
[Project > Tools > Rhubarb Lipsync TPI] - This should open the Plugin Popup Window. If it's your first time running it, you will see a error message
"Can't proceed. Rhubarb binary doesn't exist on path defined in Settings".
So click the gear icon to open the Settings window. - The Settings window should also give you a warning stating that it didn't find Rhubarb's binary at the plugin settings.
Click the highlighted link"Rhubarb Lip Sync"at the top to go to Rhubarb Lip Sync's releases page. - Download the appropriate release for your operating system.
- Extract the contents from the zip file you downloaded to your desired directory. (We'll use the plugin directory
"res://addons/rhubarb_lipsync_integration") - If the directory you extracted is inside the project directory, rename the Rhubarb Lip Sync folder to the same name but with a dot in the beggining. ex:
".rhubarb-lip-sync-1.10.0-linux"
This will make Godot ignore this folder. - Go back to Godot, click the Rhubarb Path button and call Rhubarb Lip Sync Binary from the folder you just extracted. *
- Hit save button.
*4 -
If you get a warning saying that the plugin didn't recognize the program as a binary file, try renaming Rhubarb's binary to the same extension as your Godot binary. ex: "rhubarb.32"
- Make sure the Edited Scene has the following nodes: AnimationPlayer, AudioStreamPlayer and (Sprite or AnimatedSprite).
- Select AnimationPlayer and open/create the Animation to lip sync.
- Click on
Add trackand create an Audio Playback Track - Drag the .WAV or .OGG audio file to the track
- Now Browse
[Project > Tools > Rhubarb Lipsync TPI] - Select which node to lip sync the mouth from the tabs: Sprite or AnimatedSprite
- Assign the NodePath to the mouth Sprite || Assign NodePath to AnimatedSprite and SpriteFrames animation.
- Assign the NodePath to the AnimationPlayer
- Assign the Animation to be lip synced
- Assign the NodePath to the AudioStreamPlayer
- Assign the Audio key containing the voice recording to be lip synced
- Press Done.
When all values are correctly assigned, Rhubarb Lip Sync should start generating the lip sync file. You can check the progress by opening Godot's Debug Console on Windows. For other OS's you rely on Godot printing on Editor Console.
When Rhubarb Lip Sync has finished generating lip sync data, there'll be a delay until the lip sync is imported to the Animation. That's because the plugin relies on a Timer instead of communicating with Rhubarb.
Speech Recognizers are speech recognition libraries used by Rhubarb Lip Sync to determine what phonemes or words are being said in a voice recording
By default, Rhubarb Lip Sync TPI uses pocketSphinx as the recognizer for Rhubarb Lip Sync, but you can change it to phonetic in Settings.
| Recognizer | Usecase |
|---|---|
| pocketSphinx | recommended for English voice recordings. |
| phonetic | recommended for any non-English voice recordings. |
Speech Recognizers are not used directly through this plugin, but as a parameter for executing Rhubarb Lip Sync. For more information about Recognizers, see Rhubarb Lip Sync's section on Recognizers
Mouth libraries are groups of mouth textures stored in file format that can be reused to lip sync a voice recording.
The default mouth library is located at the directory "res://addons/rhubarb_lipsync_integration/assets/lipsync default mouthshapes/"
You can create your own Mouth Library by pressing the "Plus" icon in the Mouthshape Library bar then setting a name for it. The name should be alphanumerical (a-z, 0-9) and spaces should be replaced with "_" underline.
To rename a mouth library, click the "Pen" icon in the Mouthshape library bar and setting a new name for it with the same rules as stated above.
To replace the Mouthshape texture, press the desired mouthshape image and browse to the new image filepath. (By default the plugin uses a custom FileDialog named FileSelectorPreview) Press "Save" icon in the Mouthshape Library
To remove a Mouthshape texture, press the "Delete" icon in the MouthShape Library.
Please note that Rhubarb Lip Sync TPI uses the file paths to reference the textures, so if the images change directory the texture will fail to load.
In AnimatedSprites mouthshapes are stored in frames, so there's no need to assign image file paths. By default the mouthshape frame sequence is as follows:
rest - MBP - FV - O - U - etc - E - L - AI
but it can be changed by pressing a a MouthFrameIcon and assigning a frame in a frame selector popup.
After you lip sync a lot of voice recordings, you may find your lip sync output directory to be full of used lip sync files you probably don't need anymore, so Rhubarb Lip sync TPI has a Cleaning Routine functionality to clean all ".TSV" and ".WAV" lip sync data and sliced audio files after a determined event.
These events are:
| Event | Condition |
|---|---|
| Never | Cleaning is never executed |
| OpenPlugin | Project is opened or plugin is enabled |
| ClosePlugin | Project is closed or plugin is disabled |
Godot doesn't have a native "process_died" signal so the plugin relies on checking if an output ".TSV" (lip sync data) file has been created before importing the animation.
This is done through a Timer node that calls the checking every interval of seconds for a calculated maximum period of time (This is why the plugin prints "(NOT A ESTIMATE) Plugin will wait for max...")
By default the interval is 3 sec and the maximum number of calls is 200 (this should roughly estimate 10 minutes until plugin gives up on waiting for an output). These values can be changed in the "file_checks" section of the Settings:
timer_max_calls - maximum number of file checking calls
timer_sec - the interval in seconds
Plugin Demo Video (version 1.0)
Video tutorial on how to use Rhubarb Lip Sync TPI's basic functionality (version 1.0)
FileSelectorPreview is an alternative to Godot's FileDialog. The main advantage is the presence of thumbnails and an Icon display structure. It has a zoom feature to display more/less files at once. You can also select a file by typing the full file path into the directory LineEdit.
By default FileSelectorPreview is the default FileDialog for selecting images for mouthshapes, but this can be changed at section file_selection at Settings.
file_selector_preview - Custom file dialog
godot - Godot's native FileDialog
When you have a really long audio file, but want to lip sync only a small portion, you can slice the audio key using it's start_offset and end_offset properties. However it works differently depending on the audio key sample's extension:
.WAV files will save a new sliced input file to be used as a parameter for Rhubarb Lip Sync, so only the SLICED LENGTH AUDIO will have lip sync generated. Therefore there's no data outside offsets to ignore.
.OGG files will have the lip sync for the FULL LENGTH AUDIO generated and the data outside the offsets will be ignored on importing.
Rhubarb Lipsync T.P. Integration for Godot allows you to use the functions that communicate with Rhubarb Lip Sync directly by gdscript. For that you need to call the plugin instance first. The addon uses Godot's group to easily call the plugin path from anywhere.
But please note this is experimental, there's no guarantee something will not break. This was only tested on tool scripts.
For example, create an empty scene and attach this script to a Control node:
tool
extends Control
export var call_rhubarb_lipsync_tpi_plugin :bool setget set_rhubarb_lipsync_tpi_plugin #fake button | pressed via inspector
var rlsi_plugin_group :String= "plugin rhubarb_lipsync_integration" #current plugin group name
var rlsi_pluginInstance :EditorPlugin #Rhubarb Lipsync Integration
#Called when user clicks the fake button.
func set_rhubarb_lipsync_tpi_plugin(value):
call_rhubarb_lipsync_tpi_plugin = false #fake button | value is ignored
var group :Array= get_tree().get_nodes_in_group(rlsi_plugin_group)
for node in group:
if node is EditorPlugin: #There should be only one EditorPlugin type at this group.
rlsi_pluginInstance = node
#plugin now can be called by pluginInstance.After you have called the Rhubarb Lip Sync TPI plugin you can use these functions to communicate with Rhubarb Lip Sync TPI plugin:
| Function | Description |
|---|---|
| run_rhubarb_lipsync() | Executes Rhubarb Lip Sync in Godot |
| import_lipsync() | Imports a Lipsync TSV (tab-separated-value) file into an Animation resource. |
| import_deferred_lipsync() | Imports lip sync to Animation but only after Rhubarb Lip Sync has finished generating the output lip sync file (be careful with this as it yields until Rhubarb Lip Sync finishes generating lip sync) |
| get_prestonblair_mouthtexture() | Returns a StreamTexture from a Rhubarb Lip Sync mouthshape input |
| get_mouthDB() | Returns a Dictionary with Mouthshape database from a mouthshape library |
see below for a breakthrough of the function arguments.
| argument | type | description |
|---|---|---|
| path_input_audio | String | FilePath to audio file (should be .WAV or .OGG) |
| are_paths_absolute | bool | False for a relative path ("res://") True for an absolute path ("C://") |
| length | float | Audio length in seconds |
| argument | type | description |
|---|---|---|
| anim | Animation | Animation Resource Object from animationPlayer |
| animationPlayer | AnimationPlayer | AnimationPlayer Instance |
| audiokey | Dictionary | Data about the audio key from anim |
| audioPlayer | AudioStreamPlayer | AudioStreamPlayer Instance |
| mouthDB | Dictionary | Mouthshape database |
| extraParams | Array | Parameters for the lip sync import (Sprite / AnimatedSprite, Animation) |
| argument | type | description |
|---|---|---|
| audiokey | Dictionary | Data about the audio key from anim_name's Animation |
| extraParams | Array | Parameters for the lip sync import (Sprite / AnimatedSprite, Animation) |
| audioPlayer | AudioStreamPlayer | AudioStreamPlayer Instance |
| animationPlayer | AnimationPlayer | AnimationPlayer Instance |
| anim_name | String | Animation resource's name from AnimationPlayer |
| mouthDB | Dictionary | Mouthshape database |
Requires that Rhubarb Lip Sync is run aswell, otherwise it yields forever and causes memory leak.
| argument | type | description |
|---|---|---|
| rhubarb_shape | String | Rhubarb Lip Sync mouthshape ("A", "B", "C") |
| mouthDB | Dictionary | Mouthshape texture database |
| argument | type | description |
|---|---|---|
| mouth_library | String | Section in the mouthshape_libraries.ini config file |
Rhubarb Lip Sync Third Party Integration for Godot does not have warranty for any eventual issue and bug that may break your project. It is always advisable that you backup your project. I don't assume any responsibility for any possible corruption or deletion of your files. Please notice this addon doesn't come with Rhubarb Lip Sync's binary. It's advisable you only download Rhubarb from the official Rhubarb Lip Sync Github releases page. The plugin currently has no way to determine if the file is valid.
If you encounter any errors, try opening Godot's Debug Console on Windows or start Godot from the terminal and replicate the problem. When reporting an issue, make sure to include any errors printed there. Rhubarb Lip Sync Third Party Integration for Godot