Build an AI talking avatar agent: STT, LLM, TTS, face

13 steps · about 7 h 15 min · Intermediate

  1. Using the Web Speech API (reference, MDN Web Docs, 25 min)

    Start with the browser's built-in speech recognition and synthesis to see the two ends of a voice chat before adding paid or cloud services.

  2. MediaDevices: getUserMedia() method (reference, MDN Web Docs, 15 min)

    Every voice agent page begins by asking for microphone permission and receiving a MediaStream, so learn that call and its secure-context rule.

  3. Real time communication with WebRTC (tutorial, web.dev (Google), 1 h)

    Learn how browsers stream media and data peer to peer, because the avatar's voice and video reach your page this way.

  4. Voice agents (article, LiveKit, 15 min)

    Shows the voice agent pipeline stage by stage (STT, LLM, TTS) and the latency and turn-detection ideas you need before coding.

  5. Build your first AI voice agent in Python: complete tutorial (article, LiveKit, 45 min)

    Walks through connecting STT, LLM and TTS into a working real-time conversation you can test and debug.

  6. Pipeline nodes and hooks (doc, LiveKit docs, 20 min)

    Opens up the pipeline so you can see where each stage can be inspected, filtered or replaced.

  7. Voice agents (doc, OpenAI, 20 min)

    Compares a chained STT, LLM, TTS pipeline with a single speech-to-speech model, so you can pick the right design.

  8. Getting started with the Realtime API (doc, OpenAI, 40 min)

    Builds a browser speech-to-speech agent over WebRTC and introduces interruptions (barge-in) and voice activity detection.

  9. Pipecat: Real-Time Voice & Multimodal AI Agents (repo, GitHub (Daily and community), 40 min)

    A second open-source framework that shows how swappable STT, LLM, TTS and transport services fit together, so you aren't tied to one vendor.

  10. agents/examples/avatar (repo, GitHub (LiveKit), 40 min)

    Adds the face: an official example of a voice agent with a talking-head avatar provider streamed into the call.

  11. Talking Avatar: free, open-source talking avatars for the browser (repo, GitHub (interviewflowai), 30 min)

    Shows how lip sync for a 3D avatar can be driven from any audio stream running on the user's device, which is the cheapest way to try an avatar.

  12. livekit-avatar: LiveKit voice agent with a three.js avatar (repo, GitHub (peted70), 45 min)

    Goes deeper into lip sync, emotion and gesture, showing what audio alone can't carry and how a renderer acts it out.

  13. Avatar Talk: real-time cloud-native digital human (repo, GitHub (observer04), 40 min)

    Finish by reading a full end-to-end system (browser, WebRTC room, voice agent, tools, avatar renderer) to see how the pieces deploy together.