Five demo calls won't show you where a voice agent breaks

TechCrunch once compared Coval’s approach to evaluating voice and chat agents with the way self-driving cars are tested. The comparison has a basis: the founder, Brooke Hopkins, is a former tech lead at Waymo, and the idea for Coval took shape during Y Combinator’s summer 2024 batch.

For an FDE, that idea answers the hardest question before go-live. The question is not whether the agent works. It is how many kinds of caller the agent can handle before it breaks.

A demo of five calls made by your own team cannot answer that. This is the gap Coval targets, and it is also where you will lose credibility with the client if you skip it.

What does Coval actually do?

The official documentation presents Coval as a platform for getting voice AI agents ready for real-world use. The tool runs thousands of simulated conversations before launch, with background noise, a range of voices and edge cases. According to TechCrunch, these simulations can run in parallel.

Each conversation is scored against a common set of metrics. You can add your own criteria and run regression checks, meaning you rerun the same set of scenarios after every change to see whether anything has got worse.

Coval’s homepage describes the workflow as a loop of three steps, Simulate, Observe and Review, applied both to pre-launch testing and to running evals on real production calls.

The Review step is the most notable detail. Failed conversations are passed to humans for review and, according to the docs, their feedback is used to retrain the AI judge and make the metrics more accurate.

Coval also has a native integration with Langfuse, so teams already using Langfuse for tracing can run end-to-end simulation tests across a voice agent’s entire conversation.

A worked example: a dental booking agent

Imagine you are deploying a voice agent that books appointments for a chain of dental clinics. Through customer discovery, you learn the four kinds of caller that give receptionists the most trouble: people calling from noisy places, people with strong regional accents, people who change their appointment time mid-call, and people who ask how much their insurance will cover, a question the agent must not answer on its own.

You create 50 variants of each situation, 200 conversations in total. You write three criteria as plain sentences. The agent must book the correct date and time, must transfer the call to a human when a question falls outside its scope, and must not invent an insurance figure.

Suppose the result is 170 passes and 30 failures, and 22 of the 30 failures fall into the noise group. That figure shows the fault lies in speech recognition, not in the prompt. Rewriting the prompt at this point would waste time and fix nothing.

You send the 30 failed calls to a member of clinic staff to listen back. Suppose they find 5 calls the judge failed by mistake, because the agent did in fact transfer the call at the right moment. You can now fix both the agent and the scoring. Then you rerun exactly those 200 scenarios as a regression before handover.

Limits to state plainly to the client

A simulation is only as good as the scenarios you write. If discovery missed callers reading out their insurance card number, thousands of simulated calls will not catch that bug either. That is why running evals on real production calls cannot be skipped.

The AI judge can also get it wrong. The very fact that Coval puts humans in the review loop shows the toolmaker does not treat automated scores as the final verdict. So when reporting to the client, state how many calls you checked by hand, rather than giving only a pass rate.

Finally, the docs and homepage currently centre on voice. Chat is mentioned in TechCrunch’s article and in the Y Combinator description. If the client mainly needs a chat agent, check the actual features carefully before making promises. The list of logos on the homepage, including Chime, Perplexity, ServiceNow, StubHub, Zoom, Toast and Upstart, is the vendor’s own endorsement.

What to learn before touching the tool

The first thing to learn is not Coval’s interface but the skill of turning client notes into scenarios and testable criteria. That skill works with any eval tool. It is also what separates an FDE from someone who only knows how to call an API.

Next comes the regression habit: every time you change a prompt, swap a model or switch speech-to-text provider, you rerun the same set of scenarios. Alongside it goes observability, meaning you understand a conversation’s trace well enough to tell whether a fault lies in speech recognition, in the logic or in a tool call.

When reading job descriptions, look for keywords such as “evals”, “voice agents” and “simulation”. On your CV, use numbers: how many scenarios you built, and which regressions that set caught before the client ran into them. Numbers like these show a CV reader the work you have done; a line saying “experienced with LLMs” is only a claim.

Coval can run thousands of conversations, but only someone who has sat with the clinic receptionist knows which ones are worth running.