nafyi.

Voice Agent Testing: Evaluate Before Taking Real Calls

Deepgram Learn ·

Jose Nicholas Francisco describes testing voice agents with a nine-scenario suite, scoring whole conversations, covering failure cases, and re-running the suite after prompt edits. Read 4 viewpoints with supporting evidence and source links.

Jose Nicholas Francisco
Read key viewpointsOpen Publisher (opens in a new tab)

Understand this piece

4 key points

Synthesis

  1. Start with a nine-scenario suite

    Francisco describes voice agent testing as starting with a nine-scenario suite and pass criteria that score a whole conversation.

    Supporting evidence 1

    Original excerpt

    Voice agent testing starts with a nine-scenario suite and pass criteria that score a conversation as a whole.

    Jose Nicholas Francisco · Paragraph 1

    Voice Agent Testing: Evaluate Before Taking Real Calls · Sep 9, 2026

    Context

    Launch readiness comes down to whether the agent finishes the job on a real call. One recording per scenario and a written end state give you the answer before your callers do. Re-run the suite after prompt edits.

    Read in source context →

    Continue exploring

    agent validation workflow →

  2. Score whole calls, not transcripts

    Francisco describes scoring the whole call: when the agent spoke, what it committed to, and whether the caller got what they came for. Word-level scores do not show interruptions, confirmation loops, or a confident read-back of a wrong order number.

    Supporting evidence 1

    Original excerpt

    Testing means scoring the whole call. You record when the agent spoke, what it committed to, and whether the caller left with what they came for. WER measures a transcript , so no word-level score tells you whether the agent interrupted the caller or looped on a confirmation. A confident read-back of the wrong order number also falls outside that score.

    Jose Nicholas Francisco · Paragraph 2

    Voice Agent Testing: Evaluate Before Taking Real Calls · Sep 9, 2026

    Read in source context →

    Continue exploring

    agent validation workflow →

  3. Cover real-world failure modes

    The suite covers interruptions, silence, accents, anger, and wrong numbers that happy-path demos miss.

    Supporting evidence 1

    Original excerpt

    The suite covers interruptions, silence, accents, anger, and wrong numbers that happy-path demos miss.

    Jose Nicholas Francisco · Paragraph 6

    Voice Agent Testing: Evaluate Before Taking Real Calls · Sep 9, 2026

    Read in source context →

    Continue exploring

    agent validation workflow →

  4. Re-run full test suite after every prompt edit

    Francisco says small prompt edits can cause large, model-dependent behavior changes, so every prompt edit re-runs the suite.

    Supporting evidence 1

    Original excerpt

    Small prompt edits can produce large, model-dependent behavior changes, so every prompt edit re-runs the suite.

    Jose Nicholas Francisco · Paragraph 8

    Voice Agent Testing: Evaluate Before Taking Real Calls · Sep 9, 2026

    Read in source context →

    Continue exploring

    agent validation workflow →

Key passages4

Attributed passages with the context to verify them. Open the original text to check the source.

Paragraph 1 (opens in a new tab)
agent validation workflow

Start with a nine-scenario suite

Original excerpt

Voice agent testing starts with a nine-scenario suite and pass criteria that score a conversation as a whole.
Jose Nicholas Francisco·
Context

Launch readiness comes down to whether the agent finishes the job on a real call. One recording per scenario and a written end state give you the answer before your callers do. Re-run the suite after prompt edits.

Paragraph 2 (opens in a new tab)
agent validation workflow

Score whole calls, not transcripts

Original excerpt

Testing means scoring the whole call. You record when the agent spoke, what it committed to, and whether the caller left with what they came for. WER measures a transcript , so no word-level score tells you whether the agent interrupted the caller or looped on a confirmation. A confident read-back of the wrong order number also falls outside that score.
Jose Nicholas Francisco·

Source & methodology

These viewpoints are linked to their original sources. Paraphrases are labeled and are not verbatim quotes.

Open transcript or source material (opens in a new tab)Report an issue

Explore these viewpoints by person

Related research

  • Can transcript accuracy validate a promise that a voice agent can replace a receptionist?

    A word-level transcript score does not test the whole job promised to a customer. Francisco identifies call behaviors and outcomes that it misses; Walling asks what a vendor can promise and stand behind. Mollick’s expert-led workflow also keeps review and fallback in scope. Comparing these sources clarifies the gap between measuring transcript accuracy and evaluating a replacement promise; it supplies no universal passing score.

Continue with this topic

More sources on topics discussed here. Shared topics do not imply agreement.