TalkingHead (3D)
Demo Videos
All the demo videos are real-time screen captures from a Chrome browser running
the TalkingHead test web app without any post-processing.
| Video | Description |
|---|
| Having a good hair day! – A two-part introduction to the TalkingHead's dynamic bones feature 🦴🦴 and built-in physics engine. Using custom models with rigged hair and two different hairstyles. See Appendix E for more details. |
| I chat with Jenny and Harri. The close-up view allows you to evaluate the accuracy of lip-sync in both English and Finnish. Using GPT-3.5 and Microsoft text-to-speech. |
| A short demo of how AI can control the avatar's movements. Using OpenAI's function calling and Google TTS with the TalkingHead's built-in viseme generation. |
| $$\color{transparent}{\rule{200px}{0px}}$$ | Julia and I showcase some of the features of the TalkingHead class and the test app including the settings, some poses and animations. |
Use Case Examples
Some featured videos, apps, and projects using the TalkingHead class:
| Video/App | Use Case |
|---|
| Ava. A fully in-browser avatar assistant powered by LFM3-Omni on WebGPU, Silero VAD, and TalkingHead with HeadAudio. Developed by Victor M and his team at Hugging Face. 🤗 |
| Human-AI group conversations. Researchers from UVA, Google, Northeastern, Google DeepMind, and Google Research developed DialogLab, a toolkit to author, simulate and test human-AI group conversations. 🤖🤖🤖 |
| Low-latency AI speech over WebRTC. Speech-to-speech in realtime over WebRTC using OpenAI Realtime API. Learn more about the audio-driven lip-sync module at HeadAudio.Note: The demo app uses OpenAI's gpt-realtime-2.1-mini model and requires an Ephemeral OpenAI API key. |
| Video conferencing. A video conferencing solution with real-time transcription, contextual AI responses, and voice lip-sync. The app and demo, featuring Olivia, by namnm 👍 |
| Recycling Advisor 3D. Snap a photo and get local recycling advice from a talking avatar. My entry for the Gemini API Developer Competition 2024. |
| Quantum physics using a blackboard. David introduces us to the CHSH game and explores the mystery of quantum entanglement. For more information about the research project, see CliqueVM. |
| $$\color{transparent}{\rule{200px}{0px}}$$ | Interactive Dating Profiles. ❤️ Researchers from the MIT Media Lab and Harvard used the TalkingHead class and data-driven AI to create digital twins that potential dating partners could interact with. Their paper (Baradari et al., 2025) was presented at CHI 2025 in Japan. |
Links to more projects, apps, and research using TalkingHead:
- KidsChat - A kid friendly AI chat.
- Cancer Clinical Trial Participation – Researchers at the University of Florida explored how multiple virtual agents can help overcome barriers to joining cancer clinical trials.
- TalkMateAI – Real-time Voice-Controlled 3D Avatar with Multimodal AI.
- Riverts – A platform for building, running, and analyzing interactive user-avatar conversations.
- Alter egos alter engagement – Researchers at the University of Florida used TalkingHead to explore how embodied AI chatbots can support mental well-being.
Introduction
Talking Head (3D) is a browser JavaScript class featuring a 3D avatar that can
speak and lip-sync in real-time. It also knows a set of emojis and can convert them
into facial expressions.
The class supports full-body 3D avatars (GLB) and Mixamo
animations (FBX). The avatar must have a Mixamo-compatible rig and ARKit and
Oculus viseme blend shapes. See Appendix A for details on creating your own avatar.
By default, the class uses
Google Cloud TTS for text-to-speech
and has a built-in lip-sync support for English, German, French, Brazilian Portuguese, Finnish, Lithuanian, and Bengali.
New lip-sync languages can be added by creating new lip-sync language modules.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
