Latency and quality comparison

Numbers settle debates. In this chapter, you will learn how to measure the performance of pipeline, realtime, and hybrid agents using consistent methodology. You will see real-world latency benchmarks, understand the metrics that matter for voice AI, and build a benchmarking harness that you can run against your own agents.

What you'll learn

• The key metrics for evaluating voice AI agent performance
• How to measure time-to-first-byte, end-to-end latency, and audio quality
• Real-world latency numbers across pipeline, OpenAI Realtime, and Gemini Live
• How to build a repeatable benchmarking harness

Metrics that matter

Voice AI has its own performance vocabulary. These are the metrics you should track:

Real-world latency benchmarks

The following numbers come from testing the same 50-prompt benchmark suite against each architecture, measured from a US-East data center. All models used default configurations.

Time-to-First-Byte (TTFB)

End-to-End Latency

Audio quality (MOS)

Building a benchmarking harness

To produce consistent, repeatable measurements, build a harness that sends the same prompts to each architecture and records timing data:

import time
import statistics
from dataclasses import dataclass


@dataclass
class BenchmarkResult:
  prompt: str
  ttfb_ms: float
  e2e_ms: float
  transcript: str


BENCHMARK_PROMPTS = [
  "What is your return policy?",
  "I need help with order number 12345.",
  "Can you explain the difference between your Basic and Pro plans?",
  "My package hasn't arrived yet. It was supposed to be here yesterday.",
  "Yes.",
  "Thank you, that's all I needed.",
]


def summarize_results(results: list[BenchmarkResult]) -> dict:
  ttfb_values = [r.ttfb_ms for r in results]
  e2e_values = [r.e2e_ms for r in results]

  return {
      "ttfb_p50": statistics.median(ttfb_values),
      "ttfb_p90": sorted(ttfb_values)[int(len(ttfb_values) * 0.9)],
      "ttfb_mean": statistics.mean(ttfb_values),
      "e2e_p50": statistics.median(e2e_values),
      "e2e_p90": sorted(e2e_values)[int(len(e2e_values) * 0.9)],
      "e2e_mean": statistics.mean(e2e_values),
  }


# Usage: run each prompt through your agent and record timing
# results = []
# for prompt in BENCHMARK_PROMPTS:
#     start = time.monotonic()
#     ttfb, transcript = await run_agent_prompt(agent, prompt)
#     e2e = (time.monotonic() - start) * 1000
#     results.append(BenchmarkResult(prompt, ttfb, e2e, transcript))
# print(summarize_results(results))

Cost comparison

Latency is not the only number that matters. Here is an approximate cost comparison for a 10-minute conversation with moderate back-and-forth:

What you learned

• TTFB is the most impactful metric for user experience in voice AI
• Realtime models (especially Gemini Flash) consistently deliver lower TTFB than pipelines
• Pipeline agents produce higher audio quality thanks to dedicated TTS providers
• Cost varies significantly — pipeline with a small LLM is cheapest, OpenAI Realtime is most expensive
• Repeatable benchmarking requires consistent prompts, controlled conditions, and percentile-based reporting

Next up

You have the data. In the final chapter, you will synthesize everything into a decision framework — a structured approach for choosing between pipeline, realtime, and hybrid architectures based on your specific requirements, constraints, and priorities.