/Audio2Face-3D-Samples

A service to convert audio to facial blendshapes for lipsyncing and facial performances.

Primary LanguagePython

Audio2Face-3D Microservice

NVIDIA Audio2Face-3D NIM (A2F-3D NIM) is delivering generative AI avatar animation solutions based on audio and emotion inputs.

The Audio2Face-3D Microservice converts speech into facial animation in the form of ARKit Blendshapes. The facial animation includes emotional expression. Where emotions can be detected, the facial animation system captures key poses and shapes to replicate character facial performance by automatically detecting emotions in the input audio. Additionally emotions can be directly specified as part of the input to the microservice. A rendering engine can consume Blendshape topology to display a 3D avatar's performance.

This Git repository stores resources presented in Audio2Face-3D Microservice documentation. However, Audio2Face-3D NIM can be obtained through an evaluation license of NV AI Enterprise (NVAIE) through NGC.

  1. Try out the A2F-3D NIM experience
  2. Try NIM For Digital Human

Example Workflows

For example worflows, please see the NVIDIA ACE samples, workflows, and resources Git repository.

Documentation

Full Audio2Face-3D NIM developer documentation.

Repository structure

Here is a quick overview of the available Audio2Face-3D resources in this Git repository:

  • configs/ folder:
    • Contains example static deployment, advanced and stylization configurations.
  • early_access/ folder:
    • Contains examples and sample applications for our early access releases.
  • example_audio/ folder:
    • Contains example audio files.
  • migration/ folder:
    • Contains scripts to help you migrate from previous versions of A2F-3D to latest.
  • proto/ folder:
    • Contains gRPC proto definitions for the microservices and instructions for installing them.
  • quick-start/ folder:
    • Contains docker compose files deploying Audio2Face-3D (A2F-3D) and collecting telemetry data.
  • scripts/ folder:
    • Contains example applications using Audio2Face-3D (A2F-3D) Microservice and instructions for running the scripts.

License

Github - Apache 2

ACE NIMs and NGC Microservices - NVIDIA AI Product License

Note: This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.