avatar-graph-comfyui

Wanna animate or got a question? Join our Discord

A custom nodes module for creating real-time interactive avatars powered by blender bpy mesh api + Avatech Shape Flow runtime.

WARNING
We are still making changes to the nodes and demo templates, please stay tuned.

Demo

Interact 👆	Interact 👆	Interact 👆	Interact 👆
Interact 👆	Interact 👆	Interact 👆	Interact 👆

How to?

Basic Rigging Workflow Template
Best Practices for image input
Custom Nodes List
Shape Flow
Installation
Development
Join Discord 💬

Basic Rigging Workflow Template

1. Creating an eye blink and lipsync avatar

For optimal results, please input a character image with an open mouth and a minimum resolution of 768x768. This higher resolution will enable the tool to accurately recognize and work with facial features.

Download: Save the image, and drag into Comfyui or Simple Shape Flow

2. Creating an eye blink and lipsync emoji avatar

Download: Save the image, and drag into Comfyui

Download: Save the image, and drag into Comfyui or Dog Workflow

Best practices for image input

1. Generate a new character image

We need a character image with an open mouth and enable the tool to easily recognize facial features, so please add to the prompt: looking at viewer, detailed face, open mouth, [smile], solo,eye-level angle

Download: Character Gen Template

2. Make existing character image mouth open (Inpaint)

To maintain consistency with the base image, it is recommended to utilize a checkpoint model that aligns with its style.

Download: Mouth Open Inpaint Template

Inpaint Demonstration

270589612-ff48c3d9-7292-4505-8993-8f117cee34ff.mp4

3. Pose Constraints (ControlNet)

Place normal and openpose image with reference to images.

Download: ControlNet Gen

Recommend Checkpoint Model List

Anime Style SD1.5

Realistic Style SD1.5

Custom Nodes

Expand to see all the available nodes description.

All Custom Nodes

Name	Description	Preview
`Segmentation (SAM)`	Integrative SAM node allowing you to directly select and create multiple image segment output.

Name	Description	Preview
`Create Mesh Layer`	Create a mesh object from the input images (usually a segmented part of the entire image)
`Join Meshes`	Combine multiple meshes into a single mesh object
`Match Texture Aspect Ratio`	Since the mesh is created in 1:1 aspect ratio, a re-scale is needed at the end of the operation
`Plane Texture Unwrap`	Will perform mesh face fill and UV Cube project on the target plane mesh, scaled to bounds.

Name	Description	Preview
`Mesh Modify Shape Key`	Given shape key name & target vertex_group, modify the vertex / all vertex’s transform
`Create Shape Flow`	Create runtime shape flow graph, allowing interactive inputs affecting shape keys value in runtime

Name	Description	Preview
`Avatar Main Output`	The primary output of the .ava file. The embedded Avatar View will auto update with this node's output

Shape Flow

Installation

Method 1 - Windows

Download Python environment from here
Unzip it to ComfyUI directory
Run the run_cpu_3.10.bat or run_nvidia_gpu_3.10.bat
Install avatar-graph-comfyui from ComfyUI Manager

Method 2 - macOS/Linux

Make sure your Python environment is 3.10.x as required by the bpy package. Then go to the ComfyUI directory and run:

Suggest using conda for your comfyui python environment

conda create --name comfyui python=3.10

conda activate comfyui

pip install -r requirements.txt

cd custom_nodes
git clone https://github.com/avatechgg/avatar-graph-comfyui.git
cd avatar-graph-comfyui && python -m pip install -r requirements.txt
Restart ComfyUI with enable-cors-header python main.py --enable-cors-header or (for mac) python main.py --force-fp16 --enable-cors-header

Development

If you are interested in contributing

For comfyui frontend extension, frontend js located at avatar-graph-comfyui/js

Web stack used: vanjs tailwindcss

Install deps

pnpm i

Run the dev command to start the tailwindcss watcher

pnpm dev

For each changes, simply refresh the comfyui page to see the changes.

p.s. For tailwind autocomplete, add the following to your vscode settings.json.

{
    "tailwindCSS.experimental.classRegex": [
        ["class\\s?:\\s?([\\s\\S]*)", "(?:\"|')([^\"']*)(?:\"|')"]
    ]
}

Update blender node types

To update blender operations input and output types (stored in blender/input_types.txt), run:

python generate_blender_types.py

FAQ

What is `--enable-cors-header` used for?

It is used to enable communication between ComfyUI and our editor (https://editor.avatech.ai), which is in charge of animating static characters. The only messages exchanged between them are the character data like the meshes of eyes and mouth, and the JSON format of our editor graph.

When you execute the ComfyUI graph, it sends the character data and the JSON graph to our editor for animating. When you modify and save the graph in our editor, it sends the modified graph back to ComfyUI. To validate it, you can open the js/index.js, and log the message in window.addEventListener("message", ...) and postMessage(message).

You can also run ComfyUI without the --enable-cors-header: execute the ComfyUI workflow, then download the .GLB or .GLTF format by right clicking the Avatar Main Output node and Save File option. Yet, this will disable the real-time character preview in the top-right corner of ComfyUI. Feel free to view it in other software like Blender.