The voice AI pipeline and LiveKit plugins

Every voice AI agent is built from the same three components: speech-to-text, a language model, and text-to-speech. LiveKit's Agents framework treats each component as a swappable plugin — you pick a provider, pass it to AgentSession, and the framework handles streaming, buffering, and orchestration. This chapter explains how the pipeline works, how the plugin architecture enables rapid experimentation, and when to use the traditional pipeline versus a realtime speech-to-speech model.

What you will learn

• How audio flows through the STT, LLM, and TTS stages with streaming at each step
• How LiveKit Inference provides a unified model interface for voice AI — no per-provider API keys needed
• How LiveKit's plugin architecture lets you swap any provider in one line
• The difference between pipeline mode and realtime mode (OpenAI Realtime API, Gemini Live)
• When to choose pipeline mode versus realtime mode
• How to configure AgentSession with Inference or plugin combinations

The three-stage pipeline

When a user speaks to a LiveKit voice agent, the audio passes through three stages in sequence. Because every stage streams its output, they overlap — the next stage begins processing before the previous one finishes.

The total end-to-end latency is the sum of all three stages plus WebRTC transport overhead. A well-tuned pipeline delivers first audio in 300-700ms — fast enough to feel conversational.

The latency budget

Human conversation has a natural rhythm. The average gap between turns is about 200ms. Users notice awkwardness around 500ms. Beyond one second, the interaction feels broken.

LiveKit Inference: the simplest way to get started

Before diving into individual plugins, you should know about LiveKit Inference — a unified model interface included in LiveKit Cloud. It provides access to STT, LLM, and TTS models from providers like OpenAI, Google, Deepgram, AssemblyAI, Cartesia, ElevenLabs, Rime, Inworld, xAI, DeepSeek, Groq, Cerebras, and more — all without managing separate API keys for each provider.

from livekit.agents import AgentSession, Agent, RoomInputOptions, inference

# LiveKit Inference — no per-provider API keys needed
session = AgentSession(
  stt=inference.STT(model="deepgram/nova-3", language="en"),
  llm=inference.LLM(model="openai/gpt-4.1-mini"),
  tts=inference.TTS(model="cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
)

await session.start(
  agent=Agent(instructions="You are a helpful assistant."),
  room=ctx.room,
  room_input_options=RoomInputOptions(),
)

You can also use string descriptors as a shortcut:

session = AgentSession(
  stt="deepgram/nova-3:en",
  llm="openai/gpt-4.1-mini",
  tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
)

LiveKit Inference handles billing, routing, and connection management automatically. Swapping models is as simple as changing the model string.

How the plugin architecture works

For providers not available in Inference, or when you need provider-specific features, LiveKit's Agents framework uses a modular plugin system. Each AI provider ships as a separate Python package under the livekit.plugins namespace. You import the plugin, instantiate it with your configuration, and pass it to AgentSession. The framework handles all streaming, buffering, and error recovery.

Here are all the plugin packages available:

Each plugin reads its API key from an environment variable automatically — DEEPGRAM_API_KEY, OPENAI_API_KEY, CARTESIA_API_KEY, and so on. The OpenAI plugin also supports custom base_url for OpenAI-compatible providers like Together AI, Fireworks, Cerebras, and self-hosted vLLM.

Configuring AgentSession with plugins

Here is a complete example wiring three plugins together:

from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import deepgram, openai, cartesia

session = AgentSession(
  stt=deepgram.STT(model="nova-3", language="en"),
  llm=openai.LLM(model="gpt-4.1-mini"),
  tts=cartesia.TTS(model="sonic-3", voice="warm-professional"),
)

await session.start(
  agent=Agent(instructions="You are a helpful assistant."),
  room=ctx.room,
  room_input_options=RoomInputOptions(),
)

Swapping a provider means changing one import and one line. Want Anthropic for the LLM and ElevenLabs for TTS?

from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import deepgram, anthropic, elevenlabs

session = AgentSession(
  stt=deepgram.STT(model="nova-3", language="en"),
  llm=anthropic.LLM(model="claude-sonnet-4-20250514"),
  tts=elevenlabs.TTS(model="eleven_turbo_v2", voice_id="your-voice-id"),
)

await session.start(
  agent=Agent(instructions="You are a helpful assistant."),
  room=ctx.room,
  room_input_options=RoomInputOptions(),
)

Want the cheapest possible stack? Use Gemini Flash and Deepgram Aura:

from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import deepgram, google

session = AgentSession(
  stt=deepgram.STT(model="nova-3"),
  llm=google.LLM(model="gemini-2.5-flash"),
  tts=deepgram.TTS(model="aura-2", voice="asteria"),
)

await session.start(
  agent=Agent(instructions="You are a helpful assistant."),
  room=ctx.room,
  room_input_options=RoomInputOptions(),
)

This plug-and-play design is one of the biggest advantages of LiveKit's pipeline architecture. You are never locked into a single provider, and you can A/B test different combinations by changing a few lines — or use LiveKit Inference to swap models with just a string change.

Pipeline mode vs realtime mode

LiveKit supports two fundamentally different architectures for voice AI.

Pipeline mode (STT + LLM + TTS)

This is the three-stage architecture described above. You choose a separate provider for each stage and the framework streams data between them. This is the default and most flexible approach.

Realtime mode (speech-to-speech)

OpenAI's Realtime API, Google's Gemini Live, and xAI's Grok offer speech-to-speech models that replace the entire pipeline with a single model. Audio goes in, audio comes out — no intermediate text. LiveKit supports these through dedicated realtime plugins.

from livekit.agents import AgentSession, Agent, RoomInputOptions
from livekit.plugins import openai

# OpenAI Realtime — single model replaces STT + LLM + TTS
session = AgentSession(
  llm=openai.realtime.RealtimeModel(
      model="gpt-4o-realtime-preview",
      voice="alloy",
  ),
)

await session.start(
  agent=Agent(instructions="You are a helpful assistant."),
  room=ctx.room,
  room_input_options=RoomInputOptions(),
)

Other realtime options include google.realtime.RealtimeModel for Gemini Live and xai.realtime.RealtimeModel for xAI Grok.

When to use each mode

What you learned

• The voice AI pipeline has three streaming stages: STT, LLM, and TTS, with a target latency budget of under 500ms
• LiveKit Inference provides a unified interface to models from 15+ providers — no per-provider API keys needed
• LiveKit's plugin ecosystem includes 17+ open source plugins covering all major AI providers — from OpenAI and Anthropic to Speechmatics, NVIDIA, Rime, Inworld, xAI, and more
• Configuring AgentSession with Inference or plugin combinations lets you A/B test providers trivially
• Pipeline mode offers maximum flexibility; realtime mode (OpenAI, Google, xAI) offers potentially lower latency at the cost of flexibility
• For most production use cases, pipeline mode with LiveKit Inference is the recommended starting point

Next up

Now that you understand the pipeline architecture and how plugins work, the next chapter compares every major STT, LLM, and TTS provider — with actual LiveKit plugin code for each one.