Speed you can trust: The STT metrics that matter for voice agents

For voice agents, latency only matters once a transcript is accurate and stable enough to act on. This post breaks down which speech-to-text timing metrics actually affect agent response speed, and why Pipecat’s evals are the clearest signal we’ve seen so far.

In transcription, there are lots of acronyms for lots of latencies, but, only one that truly counts for voice agents.

It’s hard to benchmark real-time latency and accuracy.

There are only a handful of public benchmarks, such as Coval.ai, before you have to resort to metrics from STT providers which always seem to end up… favorable, shall we say.

We have our own internal metrics of course, which we think are robust and fair - but why would you trust us?

The most important thing to remember? Any latency metric needs the context of accuracy.

I would rather talk to a bot that takes 100ms longer to respond but not have to repeat myself because it transcribed something wrong.

And not just accuracy: reliability and cost have to factor in when making a decision about a provider.

That's why we aren't just targeting speed - we’re targeting “Speed you can trust”.

In this post I will run you through what the real latency of a voice agent is, the various transcription ‘speeds’ I see talked about in the market and which count for voice agents. I’ll then give you a rundown of the Pipecat evaluations and why I like them so much.

Partials, finals and turns?

Just to catch everyone up on the lingo:

Partials

Partial transcripts, sometimes called ‘interim’, are the current best guess.

They are a volatile transcript subject to change.

If you are halfway through a word ‘Speech matt’ it will output its current best guess and update it as you finish speaking or as context around it changes. These are ultralow latency being emitted sometimes before you even finish a word. 

Mostly you would just use them as a visualization, and confirmation of the transcription process. But some people do use them for ‘speculative generation’ - going to the LLM before the user finishes speaking.

Finals

A final transcript is, well, final.

Once emitted, we won't change it.

Because we won't change it, we fundamentally need a longer amount of time between you saying a word and us giving it back to you as a final.

We need to make sure that you are not halfway through a word, or that the surrounding context doesn’t change what we think you have said.

These make up the ground truth that is generally used to go to the LLM.

Turns

A turn is one half of a conversation. My turn is finished when I stop speaking and expect a response.

The challenge for any system is knowing the difference between a mid-sentence breath and a finished thought.

If I say, "Add to the shopping list, a box of whatsitcalled… " and stop, I'm likely just searching for a word.

If I say, "Add to the shopping list, a box of Earl Grey tea" and then stop for 500ms, the turn is likely complete and the agent should respond.

Detecting this boundary in real-time is a specific engineering problem.

Frameworks like Pipecat and LiveKit use their own turn-detection models to decide when to "close" the user's turn and trigger the LLM.

Voice Agent latency

Also sometimes called speech-to-speech latency.

When people talk about latency of a voice agent they mean the response latency; the time from me finishing speaking to the bot starting speaking.

In that time a lot of things have to happen.

The first of which being detecting the person has actually finished speaking and getting the final transcript back from the transcription service. Then someone else can do the rest: RAG, LLM call streaming the output to a TTS service which in turn is streaming audio back to the user.

Transcription Latencies, and what contributes to voice agent latency

If you really want to get into it, you can slice and dice latency any which way you like.

Here is a subsection of the numbers I see most often, ordered from "least relevant" to "most critical" for voice agents:

RTF (Real-Time Factor)

First off, this is not a latency. This is what you might call ‘speed’.

It's not really relevant to real-time applications at least from a user perspective. But it is a number that shows up a lot on batch benchmark sites.

This is how long it takes to transcribe a second of audio (on average). So if you send us a 100 second audio file and we process it and give the transcript back to you in 5 seconds we would have an RTF of 0.05. Lower is better.

As I say this has very little to do with the user experience of real-time transcription and voice agents latency.

As long as this is less than 1 (i.e. real time) all is good. Artificial Analysis measure what they call a Speed factor for a 10 minute section of audio. Speed factor is the reciprocal of RTF (higher is better). They measure a Speed Factor of 43.9 for our standard and 24.1 for our Enhanced models. Although this will vary by file size.

TTFB (Time To First Byte)

Sometimes known as TTFT (Time to First Token/Transcript). In my opinion, this is the worst metric in the industry, but I see it cited over and over again.

This is the delay from the moment you start streaming audio to the moment you get the very first piece of transcript back.

There are a few things wrong with this. Primarily: how long is a word? If I say "Supercalifragilisticexpialidocious," an STT engine is well within its rights to wait until I’ve finished the mouthful to ensure the transcript is accurate. That results in a "long" TTFB. Alternatively, an engine could fire back "Super" almost straight away. On paper, that's "faster," but it hasn't given the agent any useful context yet.

Returning a fragment early isn't necessarily a bad thing, but consider the case of Deepgram Flux as reported by the benchmarks at Coval.ai.

If you look at the data, they almost always return the word "Two" first, regardless of what is actually being said. It’s likely just a statistical bias toward common first words in a dataset. They return it incredibly quickly but it’s not meaningful. Firing back a volatile guess doesn't make a service fast.

It just means the rest of your AI stack has to wait for a correction anyway before it can actually do anything.

STT ≠ TTS (or LLMs)

TTFB is popular because it’s a standard way to measure LLMs and TTS systems.

It's like there is a natural parity between the stacks, but that is silly. The best argument for it is that it's a proxy for the rest of the latency picture, but that is also silly. It's just easy to measure. You send in some audio and see how long it takes for something - anything - to come back. In transcription, the first byte is often just a guess that might change a millisecond later. It’s not a reliable indicator of when your agent can actually start working.

Head over to the STT benchmarks at Coval, to find independent TTFB measurements, but pay attention to our WER and what others send back in their first byte.

Partials Latency

This is sometimes called ‘word emission latency’.

I’m talking about the time from me finishing a word to getting the first partial that relates to it. Note that the partial could be right or it could be wrong. As with all these latencies, there is an accuracy associated with all of them.

It's also worth noting Partial Update Cadence - this is the rate at which Partials are changed after they first arrive.

Partial latency is probably what a lot of people ‘feel’ when using a transcription service with written output. Even if it’s a partial and sometimes wrong or halfway through a word, the partials give you real-time feedback as you talk. Generally, the quicker that happens, the happier the user is.

For voice agents, partials could potentially be used for speculative generation. You go to the LLM ahead of me finishing a turn so that as soon as I finish speaking, you are ready to go.

Similarly, you could use the partials to do semantic turn detection - does this sentence look like it's finished?