Understanding AgentSession

In the previous chapter, you built a working agent with default model configuration. In this chapter, you will take explicit control of the voice pipeline by choosing your STT, LLM, and TTS models, setting up voice activity detection, and learning about the events that let you react to everything happening in a conversation.

What you will learn: The AgentSession lifecycle, how to configure each model in the pipeline, how VAD works, and the key events you can listen for.

What you will build: An upgraded dental receptionist with Deepgram STT, GPT-4o-mini, Cartesia TTS, and Silero VAD — all explicitly configured.

The AgentSession lifecycle

Every conversation your agent has follows a predictable lifecycle. Understanding it helps you know when to configure things, when to react, and when cleanup happens.

Configuring models explicitly

In Chapter 2, you called session.start() without specifying models. The framework used sensible defaults. Now let us take explicit control. Each stage of the pipeline — STT, LLM, TTS — is configured with a plugin.

from livekit.agents import AgentServer, rtc_session, Agent, AgentSession
from livekit.plugins import openai, silero, deepgram, cartesia

server = AgentServer()


@server.rtc_session
async def entrypoint(session: AgentSession):
  await session.start(
      agent=Agent(
          instructions=(
              "You are a friendly receptionist at Bright Smile Dental clinic. "
              "Greet callers warmly and help them with appointment inquiries. "
              "Keep your responses short and conversational — one to two sentences at most."
          ),
      ),
      room=session.room,
      stt=deepgram.STT(model="nova-3"),
      llm=openai.LLM(model="gpt-4o-mini"),
      tts=cartesia.TTS(voice="79a125e8-cd45-4c13-8a67-188112f4dd22"),
  )


if __name__ == "__main__":
  server.run()

Three new parameters: stt, llm, and tts. Each one instantiates a plugin with model-specific configuration. Let us examine each.

Speech-to-Text: Deepgram Nova-3

stt=deepgram.STT(model="nova-3")

Deepgram Nova-3 is a streaming STT model optimized for real-time transcription. "Streaming" means it produces partial transcripts as audio arrives — it does not wait for the user to finish speaking. This is critical for low latency: the LLM can begin processing before the user completes their sentence.

You can also configure language, keywords, and other Deepgram-specific options:

stt=deepgram.STT(
  model="nova-3",
  language="en",
  keywords=[("Bright Smile Dental", 3.0), ("Dr. Chen", 2.0)],
)

The keywords parameter boosts recognition of domain-specific terms. The number is a weight — higher means stronger boost. This is useful for proper nouns that STT models often misrecognize.

Large Language Model: GPT-4o-mini

llm=openai.LLM(model="gpt-4o-mini")

GPT-4o-mini is a fast, capable model well-suited for voice agents. It balances quality and speed — the first token arrives quickly, which is critical when every millisecond counts. For production dental receptionists, this model handles appointment inquiries, FAQ responses, and conversational flow with ease.

You can swap in other models: gpt-4o for more complex reasoning, or use different providers entirely — anthropic.LLM(model="claude-sonnet-4-20250514") if you prefer Claude.

Text-to-Speech: Cartesia

tts=cartesia.TTS(voice="79a125e8-cd45-4c13-8a67-188112f4dd22")

Cartesia provides low-latency streaming TTS with natural-sounding voices. The voice parameter is a voice ID from the Cartesia voice library. Each voice has distinct characteristics — warmth, pace, pitch, accent.

You can also configure speech speed and emotion:

tts=cartesia.TTS(
  voice="79a125e8-cd45-4c13-8a67-188112f4dd22",
  speed="normal",
  emotion=["positivity:high", "curiosity:medium"],
)

Voice Activity Detection (VAD)

Voice Activity Detection determines when the user is speaking and when they have stopped. It is the foundation of turn-taking — the agent needs to know when to listen and when to respond.

from livekit.plugins import silero

# In your entrypoint, before session.start():
await session.start(
  agent=Agent(instructions="..."),
  room=session.room,
  stt=deepgram.STT(model="nova-3"),
  llm=openai.LLM(model="gpt-4o-mini"),
  tts=cartesia.TTS(voice="79a125e8-cd45-4c13-8a67-188112f4dd22"),
  vad=silero.VAD.load(),
)

Silero VAD is a lightweight neural network that classifies audio frames as speech or non-speech. It runs locally alongside your agent — no network call required.

Turn detection

Turn detection builds on top of VAD to decide when the user has actually finished their turn and the agent should respond. The default turn detection uses VAD silence duration — if the user is silent for a threshold (typically around 300-500ms), the agent begins its response.

from livekit.agents import AgentServer, rtc_session, Agent, AgentSession, MultimodalModel

# For more sophisticated turn detection using the LLM itself:
await session.start(
  agent=Agent(instructions="..."),
  room=session.room,
  turn_detection=MultimodalModel(),
)

MultimodalModel uses the LLM to help determine when the user is done speaking. It considers context, not just silence, which reduces false turn detections. You will explore this in depth in Chapter 9.

Key events

The AgentSession emits events throughout the conversation. These are your hooks for monitoring, logging, and reacting to what happens during a call.

from livekit.agents import AgentServer, rtc_session, Agent, AgentSession
from livekit.plugins import openai, silero, deepgram, cartesia

server = AgentServer()


@server.rtc_session
async def entrypoint(session: AgentSession):
  @session.on("agent_state_changed")
  def on_agent_state(state: str):
      print(f"Agent state: {state}")  # "listening", "thinking", "speaking"

  @session.on("user_state_changed")
  def on_user_state(state: str):
      print(f"User state: {state}")  # "speaking", "listening"

  @session.on("user_input_transcribed")
  def on_transcript(transcript):
      print(f"User said: {transcript.text}")

  @session.on("conversation_item_added")
  def on_item(item):
      print(f"New conversation item: {item}")

  await session.start(
      agent=Agent(
          instructions=(
              "You are a friendly receptionist at Bright Smile Dental clinic. "
              "Greet callers warmly and help them with appointment inquiries. "
              "Keep your responses short and conversational — one to two sentences at most."
          ),
      ),
      room=session.room,
      stt=deepgram.STT(model="nova-3"),
      llm=openai.LLM(model="gpt-4o-mini"),
      tts=cartesia.TTS(voice="79a125e8-cd45-4c13-8a67-188112f4dd22"),
      vad=silero.VAD.load(),
  )


if __name__ == "__main__":
  server.run()

Let us walk through each event:

The complete upgraded agent

Here is your dental receptionist with all the configuration from this chapter combined:

from livekit.agents import AgentServer, rtc_session, Agent, AgentSession
from livekit.plugins import openai, silero, deepgram, cartesia

server = AgentServer()


@server.rtc_session
async def entrypoint(session: AgentSession):
  @session.on("user_input_transcribed")
  def on_transcript(transcript):
      print(f"Caller said: {transcript.text}")

  await session.start(
      agent=Agent(
          instructions=(
              "You are a friendly receptionist at Bright Smile Dental clinic. "
              "Greet callers warmly and help them with appointment inquiries. "
              "Keep your responses short and conversational — one to two sentences at most."
          ),
      ),
      room=session.room,
      stt=deepgram.STT(
          model="nova-3",
          language="en",
          keywords=[("Bright Smile Dental", 3.0)],
      ),
      llm=openai.LLM(model="gpt-4o-mini"),
      tts=cartesia.TTS(voice="79a125e8-cd45-4c13-8a67-188112f4dd22"),
      vad=silero.VAD.load(),
  )


if __name__ == "__main__":
  server.run()

Test the difference

Run your updated agent:

python agent.py dev

Open the Playground, connect, and try these tests:

Try saying: "Hello, I'm calling about an appointment."

Listen to the voice. This is the Cartesia voice you configured. It should sound different from the default TTS you heard in Chapter 2 — more natural, with a distinct character.

Try saying: "I was referred by Dr. Chen at Bright Smile Dental."

Watch the terminal output. The user_input_transcribed event handler prints the transcript. Check whether "Bright Smile Dental" and "Dr. Chen" are transcribed correctly — keyword boosting should help.

Try saying: "Actually, never mind." (while the agent is speaking)

This tests interruption handling. With VAD active, the agent should detect that you started speaking and stop its current response. The agent_state_changed event will show the transition from speaking back to listening.

What changed from Chapter 2

Your agent went from implicit defaults to explicit configuration:

• STT: Default became Deepgram Nova-3 with keyword boosting
• LLM: Default became GPT-4o-mini (explicitly chosen)
• TTS: Default became Cartesia with a specific voice
• VAD: Added Silero VAD for voice activity detection
• Events: Added transcript logging to observe the pipeline

The agent's behavior is the same — greet callers and help with appointments — but the quality of each stage is now under your control. In the next chapter, you will write proper instructions that transform this basic receptionist into a polished, professional agent.