Custom TTS with realtime models

Realtime models give you low latency and natural speech comprehension — but they lock you into a handful of built-in voices. Pipelines give you any voice from any TTS provider — but add latency from three model hops. What if you could get the best of both? LiveKit supports exactly this: use a realtime model for speech understanding and reasoning, but route its text output through a dedicated TTS for speech synthesis. This is called the half-cascade architecture, and it is one of the most practical patterns for production voice agents.

What you'll learn

• How the half-cascade architecture works and why it exists
• Configuring a realtime model for text-only output
• Pairing a realtime model with any TTS provider
• When half-cascade is the right choice vs pure realtime or full pipeline

The problem: voices vs latency

In Chapters 3 and 4, you saw that realtime models offer a small set of built-in voices — six from OpenAI, five from Gemini. For many applications, that is not enough. You might need a specific brand voice, a cloned voice, fine-grained emotion control, or a provider like Cartesia or ElevenLabs with hundreds of voice options.

The obvious solution is to use a pipeline — but then you lose the realtime model's advantages: lower latency on speech comprehension, audio-native understanding of tone and emotion, and built-in turn detection that uses semantic context.

The half-cascade gives you a middle path.

How half-cascade works

Instead of sending audio to and from the realtime model, you configure it to output text only. The realtime model still receives raw audio input — it still "hears" the user with all the tonal and emotional context that provides. But instead of generating audio output, it generates text. That text is then routed to a dedicated TTS of your choice.

Compare this to the three architectures you already know:

Implementation

The implementation is straightforward. Set the realtime model's modalities to ["text"] so it produces text instead of audio, and provide a tts parameter with any TTS instance.

from livekit.agents import AgentSession, Agent, AgentServer, rtc_session
from livekit.plugins.openai import realtime
from livekit.plugins import cartesia

server = AgentServer()


@server.rtc_session
async def entrypoint(session: AgentSession):
  await session.start(
      agent=Agent(
          instructions=(
              "You are a brand ambassador for Luxe Cosmetics. "
              "Speak warmly and enthusiastically about products. "
              "Keep responses concise — two sentences maximum."
          ),
      ),
      room=session.room,
      # Realtime model handles audio input and reasoning
      # but outputs TEXT only — no built-in voice synthesis
      llm=realtime.RealtimeModel(
          model="gpt-4o-realtime-preview",
          modalities=["text"],
      ),
      # Cartesia handles all speech output with your chosen voice
      tts=cartesia.TTS(voice="your-brand-voice-id"),
  )


if __name__ == "__main__":
  server.run()

import { AgentSession, Agent, defineAgent, type RtcSession } from "@livekit/agents";
import { OpenAIRealtime } from "@livekit/agents-plugin-openai";
import { CartesiaTTS } from "@livekit/agents-plugin-cartesia";

export default defineAgent({
entry: async (session: RtcSession) => {
  await session.start({
    agent: new Agent({
      instructions:
        "You are a brand ambassador for Luxe Cosmetics. " +
        "Speak warmly and enthusiastically about products. " +
        "Keep responses concise — two sentences maximum.",
    }),
    room: session.room,
    // Realtime model handles audio input and reasoning
    // but outputs TEXT only — no built-in voice synthesis
    llm: new OpenAIRealtime({
      model: "gpt-4o-realtime-preview",
      modalities: ["text"],
    }),
    // Cartesia handles all speech output with your chosen voice
    tts: new CartesiaTTS({ voice: "your-brand-voice-id" }),
  });
},
});

The key is modalities=["text"]. This single parameter changes the realtime model from an end-to-end speech model into a "speech-in, text-out" model. The framework detects that TTS is configured and routes the text output through it automatically.

Using any TTS provider

Because the TTS is now a separate component, you can use any provider LiveKit supports — the same providers available in a full pipeline.

from livekit.plugins import cartesia, elevenlabs, google
from livekit.plugins.openai import realtime

# Option 1: Cartesia — hundreds of voices, fine-grained emotion control
llm = realtime.RealtimeModel(model="gpt-4o-realtime-preview", modalities=["text"])
tts = cartesia.TTS(voice="your-voice-id")

# Option 2: ElevenLabs — voice cloning, multilingual
llm = realtime.RealtimeModel(model="gpt-4o-realtime-preview", modalities=["text"])
tts = elevenlabs.TTS(voice="your-cloned-voice-id")

# Option 3: Google Cloud TTS — affordable, production-grade
llm = realtime.RealtimeModel(model="gpt-4o-realtime-preview", modalities=["text"])
tts = google.TTS(voice="en-US-Journey-F")

import { OpenAIRealtime } from "@livekit/agents-plugin-openai";
import { CartesiaTTS } from "@livekit/agents-plugin-cartesia";
import { ElevenLabsTTS } from "@livekit/agents-plugin-elevenlabs";

// Option 1: Cartesia — hundreds of voices, fine-grained emotion control
const llm = new OpenAIRealtime({
model: "gpt-4o-realtime-preview",
modalities: ["text"],
});
const tts1 = new CartesiaTTS({ voice: "your-voice-id" });

// Option 2: ElevenLabs — voice cloning, multilingual
const tts2 = new ElevenLabsTTS({ voice: "your-cloned-voice-id" });

Gemini Live half-cascade

The same pattern works with Gemini Live. Use text-only output modality and pair it with a TTS:

from livekit.agents import AgentSession, Agent, AgentServer, rtc_session
from livekit.plugins.google import realtime as google_realtime
from livekit.plugins import elevenlabs

server = AgentServer()


@server.rtc_session
async def entrypoint(session: AgentSession):
  await session.start(
      agent=Agent(
          instructions="You are a visual shopping assistant. "
          "Describe products the user shows on camera. "
          "Keep descriptions brief and enthusiastic.",
      ),
      room=session.room,
      llm=google_realtime.RealtimeModel(
          model="gemini-2.0-flash",
          modalities=["TEXT"],
      ),
      tts=elevenlabs.TTS(voice="your-voice-id"),
  )


if __name__ == "__main__":
  server.run()

With Gemini Live, this is especially powerful: you get multimodal input (audio + video) combined with any voice you want for output. The model sees the user's camera, hears their voice with full emotional context, reasons about both, and responds through a premium TTS voice.

Latency impact

Half-cascade sits between pure realtime and full pipeline in latency:

The additional latency comes from TTS synthesis — the realtime model's text output must be converted to audio by a separate model. But you skip the STT stage entirely, so half-cascade is still meaningfully faster than a full pipeline.

When to use half-cascade

Half-cascade is the right choice when:

• You need a specific voice but want faster-than-pipeline latency. Your brand requires a particular voice or you use voice cloning, but 500-1200ms pipeline latency is too slow.
• You want audio-native input understanding with voice control. The realtime model hears tone and emotion — and you want a premium TTS for the output.
• You need the say method for scripted speech. Pure realtime models cannot reliably produce scripted output. With a separate TTS, you can use say() to speak exact text when needed (greetings, legal disclaimers, hold messages).
• You need to load conversation history reliably. Some realtime models (notably OpenAI) can become text-only after loading extensive history. Using a separate TTS from the start avoids this failure mode entirely.
• You want Gemini's vision with a better voice. Gemini Live's built-in voices may not match your needs, but its multimodal input is unmatched. Half-cascade lets you pair Gemini's vision with any TTS.

Half-cascade is NOT the right choice when:

• Pure realtime latency is essential. If every millisecond counts, the extra TTS hop matters.
• The built-in voices are good enough. If one of the six OpenAI voices or five Gemini voices works for your application, pure realtime is simpler and faster.
• You need custom STT accuracy. Half-cascade still relies on the realtime model's built-in speech recognition. If you need a specific STT model (for domain-specific vocabulary, accent handling, or compliance), use a full pipeline.

Comparing all four architectures

You now know four distinct architectures. Here is a summary:

What you learned

• The half-cascade architecture pairs a realtime model (audio in, text out) with a dedicated TTS
• Setting modalities=["text"] converts a realtime model to text-only output while preserving audio input
• Half-cascade gives you unlimited voice selection with faster-than-pipeline latency
• The pattern works with OpenAI Realtime, Gemini Live, and other providers that support text-only modality
• Half-cascade is especially powerful with Gemini Live: multimodal input plus any voice

Next up

You have now seen four architectures: pipeline, pure realtime, half-cascade, and (up next) hybrid approaches that switch between architectures dynamically. In the next chapter, you will build agents that combine multiple architectures in a single conversation, routing to the right one based on what the user needs.