Introducing Agent STT: built for the words that cost you the call

When a voice agent fails in production, the post-mortem usually runs the same way. The prompt looks correct. The tool call was correct given the input. The response was fine.

Then someone opens the transcript. They find that the caller said "postcode is NE6 5AT" and the agent received "postcode any six five eighty". 

Every component after that point did its job. The framework took "any six five eighty" at face value, passed it to the address lookup, got no match, and asked the caller to repeat themselves. 

That's the bit we build for. Speechmatics has spent more than 15 years on speech recognition, and Agent STT is our API built specifically for the voice agents everyone's now racing to ship. We provide the layer that turns a caller's voice into something your LLM can actually trust.

Two years ago the hard problems in voice AI were orchestration, latency budgets, keeping an LLM on task. Frameworks like Pipecat and LiveKit will hand you a working pipeline in an afternoon now, and frontier models hold a conversation and reply inside a human acceptable latency.

Speech-to-text models were still being built as general purpose, real-time models and expecting these frameworks to integrate against non-optimal APIs.

Our work on Agent STT with its first model, Linden, started with an uncomfortable finding: the industry's standard accuracy scores couldn't tell us which agents were actually going to fail once they went live.

The error that matters is not the average error

Speech recognition has been graded the same way for a decade: word error rate (WER) - the share of words a model gets wrong across a test set, with every insertion, deletion or substitution weighted the same. Those test sets are mostly clean, read-aloud speech, not the overlapping mess of a real phone call.

The problem is that WER can't tell the difference between an error that costs you nothing and one that costs you the whole interaction. 

Drop "actually" from "actually I'd like the other one please" and nothing changes, this is still recoverable by the LLM. 

Lose the "t" off "I can't make the payment this month" - the first thing an 8kHz line drops - and the agent books a payment that's going to bounce.

Mishear one character in an account number, "483921" arrives as  "4A3921" and there’s nothing downstream that can recover it. 

WER scores all three the same. Real conversations are affected quite differently.

Short turns are the other blind spot. "Yep." is nothing to a human ear, but plenty of STT models trip over it, and it's often the one word an agent is sitting there waiting for. Miss it and the agent just stalls. Models trained on long-form audio lean on later context to resolve anything ambiguous - on a one-word confirmation, there is no later context coming. Linden is built to catch these.

Now add in a speaker with an accent, run it over a low-quality telephony connection, add a café in the background, and a leaderboard position stops telling you much about how an agent will actually behave on your calls.

A few benchmarks - Pipecat's among them - have started scoring "Semantic WER" instead: asking an LLM whether the meaning survived, not just whether the words matched. That's a much better proxy for what actually breaks an agent.

Linden the model underneath it

Agent STT runs on a new model, Linden, built for agents that have milliseconds to work out what was said and who said it. A full spoken segment has to come back in under 350ms - not much time to be right. Linden holds the highest accuracy we've measured across 55+ languages at that speed.

Optimizing for WER alone doesn't get you the best agent, though, so we went after the specific things that actually break voice agents in practice.

• Custom vocabulary. Add up to 1,000 words the model doesn't already know, company names, product names, part numbers, whatever's specific to your domain.

• Alphanumerics got their own treatment. Digits and letters read aloud don't follow the statistics of natural language, so the same language-model priors that help everywhere else start working against you. A model that's learned English well will happily "correct" a spoken sort code into something more probable, but wrong. Linden 1 came out of targeted improvement work on exactly this: phone numbers, account numbers, addresses, and more to come later this year.

• Highest accuracy still despite reduced latency of less than 350ms. Median finalisation is now ~350ms, at highest accuracy of 1.05% pooled semantic WER, for both the words and for who spoke them. Automatic routing to the nearest of our regional servers using our “global” endpoint saves another 50–80ms that no benchmark captures. If you've got data residency requirements, you can pin to a region (US, Europe, Australia) instead. Latency gets most of the attention in this industry, but when we asked our own users building voice agents, they ranked accuracy above it, and it's the reason most of them picked us in the first place. A fast wrong answer just means asking the caller to repeat themselves which is slower than being right the first time.

• Speaker attribution happens live along with Speaker id at no extra cost. Diarization tags every word with a speaker in real time, with no limit on how many people are in the room. Speaker ID goes a step further and lets the agent learn specific voices, so it knows exactly who it's talking to.

• The API gives you the conversation, not just words. A general-purpose STT API leaves you to build turn detection and compile segments yourself: decide when to hand off to the LLM, and how to squeeze the latency down. Agent STT was built for fast turn-based interaction from the start, which is why early customers using it have gone from zero to integrated in under an hour. 

You can also use it through LiveKit, or Pipecat, which already have it integrated.

The benchmark: Nothing is faster and more accurate

Our own measurement puts end-of-speech to finals at less than 300ms. Treat that the way you would treat any vendor's number about itself, which is why it matters that external benchmarks track us.

Pipecat, for example, takes a fixed set of 1,000 utterances, scored for semantic accuracy as well as speed, reported as a median. Linden comes in at 369ms and 1.05% semantic error (Pooled WER), which puts it on the Pareto frontier. Of the 23 streaming models on that board, nothing is both faster and more accurate.

For comparison Deepgram Nova 3 returns a final transcript around 120ms sooner, but at roughly double our error rate. The gap is quite small and it's up to developers to decide what matters for their use case. 100ms faster or more robust agent interactions.

Source: Pipecat

Linden lands on the Pipecat Pareto frontier. Of the 23 streaming models on the board, nothing is both faster and more accurate: 369ms to a final transcript, at 1.05% pooled semantic error (Pooled WER weights the scoring to the number of words in each tested file.)

Useful as public benchmarks are, it is only where an evaluation starts, not where it ends. Public benchmark guides you on where providers sit. It tells you very little about what holds up, and what doesn't, at ten thousand calls a day. Your own traffic is the test that counts: strong accents, the 8kHz path your telephony provider actually gives you, background noise, one-word confirmations, and the alphanumeric strings that trigger real actions.

Implementation

You can find a full implementation guide in our docs. 

Either use our integrations, Python SDK or the raw WebSocket API with your favourite coding agent.

Plugins for Pipecat and LiveKit Agents have been updated for today’s launch, and if you are on LiveKit Inference you have been able to select Linden there since August. For the API simply open a WebSocket at /v2/agent and set the model to linden-1. Stream your agent audio input to Speechmatics, and receive transcript segments and conversational events back on the same connection.

Full reference is in the Speechmatics docs, with runnable examples in the GitHub Academy.

Pricing

Agent STT costs less per hour than the providers it competes with. Launch pricing is $0.30 per hour, with further discounts available for high volume usage. This is one-third cheaper than Assembly, and ±10% lower than Deepgram at the list price. But remember, with discounts our price goes down to $0.16.

Competitor public prices as listed in early September 2026. If you have high volume use cases, do speak to our team for bespoke deals.

What ships next

We're always chasing the next hard problem in voice. So multilingual AI is our next focus, and it's worth separating from language coverage. 55+ languages means you no longer need to pick the one your caller speaks, it’s detected automatically. The model will also be able to follow a caller who switches languages mid-sentence, and labels the language for whatever's downstream in your stack.

Start building

Get your API key or start building with:

• LiveKit

• Pipecat

FAQ

What is Agent STT? A speech-to-text API built for production voice agents, powered by Linden, our latest model. It returns transcription, structured conversational signals, speaker attribution and custom vocabulary over a single connection.

How fast is Linden? End-of-speech finalization is less than 350ms mean and ~440ms at P95 in our internal measurements.

Which languages are supported? 55+ at launch, including bilingual and quadlingual models, with particular strength in global English accents, Arabic (including Saudi and Egyptian accents, with code-switching), Nordic languages, Japanese, and Central and Eastern European speech. Fully multilingual agents arrive with Linden 2.

Does it handle end-of-turn detection? Agent STT gives you speech and segment events to drive your own turn logic, or your framework's. Currently it’s based on speaker pauses, smart turn detection coming soon, speak to us for early access.

How much does it cost? $0.30/hr, with volume pricing available on top enabling pricing down to $0.16/hr.

What does it integrate with? Available today in LiveKit Inference and LiveKit Agents, Pipecat. If you need another framework, use our API directly or speak to us for support integrating.