Build an AI talking avatar agent: STT, LLM, TTS, face
13 steps · about 7 h 15 min · Intermediate
- Using the Web Speech API (reference, MDN Web Docs, 25 min)
Start with the browser's built-in speech recognition and synthesis to see the two ends of a voice chat before adding paid or cloud services.
- MediaDevices: getUserMedia() method (reference, MDN Web Docs, 15 min)
Every voice agent page begins by asking for microphone permission and receiving a MediaStream, so learn that call and its secure-context rule.
- Real time communication with WebRTC (tutorial, web.dev (Google), 1 h)
Learn how browsers stream media and data peer to peer, because the avatar's voice and video reach your page this way.
- Voice agents (article, LiveKit, 15 min)
Shows the voice agent pipeline stage by stage (STT, LLM, TTS) and the latency and turn-detection ideas you need before coding.
- Build your first AI voice agent in Python: complete tutorial (article, LiveKit, 45 min)
Walks through connecting STT, LLM and TTS into a working real-time conversation you can test and debug.
- Pipeline nodes and hooks (doc, LiveKit docs, 20 min)
Opens up the pipeline so you can see where each stage can be inspected, filtered or replaced.
- Voice agents (doc, OpenAI, 20 min)
Compares a chained STT, LLM, TTS pipeline with a single speech-to-speech model, so you can pick the right design.
- Getting started with the Realtime API (doc, OpenAI, 40 min)
Builds a browser speech-to-speech agent over WebRTC and introduces interruptions (barge-in) and voice activity detection.
- Pipecat: Real-Time Voice & Multimodal AI Agents (repo, GitHub (Daily and community), 40 min)
A second open-source framework that shows how swappable STT, LLM, TTS and transport services fit together, so you aren't tied to one vendor.
- agents/examples/avatar (repo, GitHub (LiveKit), 40 min)
Adds the face: an official example of a voice agent with a talking-head avatar provider streamed into the call.
- Talking Avatar: free, open-source talking avatars for the browser (repo, GitHub (interviewflowai), 30 min)
Shows how lip sync for a 3D avatar can be driven from any audio stream running on the user's device, which is the cheapest way to try an avatar.
- livekit-avatar: LiveKit voice agent with a three.js avatar (repo, GitHub (peted70), 45 min)
Goes deeper into lip sync, emotion and gesture, showing what audio alone can't carry and how a renderer acts it out.
- Avatar Talk: real-time cloud-native digital human (repo, GitHub (observer04), 40 min)
Finish by reading a full end-to-end system (browser, WebRTC room, voice agent, tools, avatar renderer) to see how the pieces deploy together.