99% accurate!? A sceptics guide to assessing speech-to-text accuracy

The quest for real world accuracy

We recently added a new open test set to our evaluation suite which covers many languages and ran the numbers to compare ourselves against some others. For this, we used open, publicly available data (called the FLEURS dataset). 

We were very happy with the results 😊:

• We performed really well in languages that are typically underrepresented, for example, we were on average 8.45% more accurate in Mandarin, and 4.77% better than our nearest competitor.

• More accurate than Amazon, AssemblyAI and Deepgram in every language we offer.

• When comparing every major ASR vendor in every language, Speechmatics is most accurate 93.73% of the time.

So far, so good 💪

What followed was a discussion about what these results meant, and whether we should plaster the full test results on our website. We haven't, and for good reason. Whilst the FLEURS test data is interesting (more on that later), we think there's an important nuance to discuss how you should judge any speech-to-text technology, and how important it is to get as close as possible to real-world, messy data.

We felt this discussion was worth sharing because, for us, it's even more important to be transparent and helpful than to massage figures to show that we come out on top. We know it can be difficult to make sense of the various claims and figures that companies in our space throw out and make the right decision for your business.

It also provides insight into how hard it is to come up with a single figure that shows in objective terms how 'good' or 'bad' a transcription provider is at actually transcribing media you give them.

So here we'll talk you through how these figures are calculated, and how sometimes keeping it simple can lead to an incomplete picture (and potentially some sub-par buying decisions).

The trouble with testing

Companies that provide speech-to-text (otherwise known as ASR, or STT) services, will often test their transcription using a dataset. These datasets are labeled, which means that a human has painstakingly listened to them (often multiple times) and written a transcript (for example medical transcription) of the audio in question (this is considered the gold standard). The test then compares this manually transcribed audio with the output of their own product. You then can measure the success or failure of this in two distinct but closely related ways:

• Word Error Rate (WER) – what percentage of words did the service get wrong? So, a WER of 33% would mean that one in every three words was transcribed incorrectly.

• Accuracy – what percentage of words does the service get right? So, an accuracy figure of 66% would mean two out of three words were transcribed correctly (and the eagle-eyed will notice that Accuracy is simply 100% - WER).

For our tests, we used the FLEURS dataset. You can read more about it here, but in short, it's a good dataset for testing your accuracy across a wide range of languages. For any company offering a broad range of languages and translation services, this dataset is a great initial way to see how they are doing.

So far, so sensible. It's good to use attributed, publicly available data as this means that you're not hiding your evaluation data and in theory, anyone could replicate the results (yay for science).

As a side note – look out for any company that doesn't show (or 'attribute') their data, since you have no idea what's in that dataset. Since you don't know this, you (or anyone else) cannot replicate the results and therefore should be very dubious of them.  

But there's a big drawback. These datasets are simply too artificial, too clean, too good. The FLEURS dataset is great in that it contains a lot of audio from different speakers across a range of languages. But they are all speakers reading from Wikipedia into a microphone. If you listen to the samples, you can hear people reading at a relatively slow and steady pace, reading well-written and formatted informational text.  Why is this a problem? Well, if you're trying to make products that use speech data in the real world, chances are you won't have the luxury of every piece of speech being this, well, clean. For example:

• People talking over each other as in regular conversations, phone calls and podcasts. 

• People speaking 'as they are thinking' – um, ers, ooh, incomplete words, half-finished thoughts 

• Noisy background, for example in a contact center obscuring an agents voice 

• Noisy background from Sporting events (thinking of trackside F1 presenters here) 

• Low-quality audio 

• Low volume audio

Any machine learning product or model designed only to excel at some of these publicly available datasets will fall down when you actually want to use them to be valuable in the real world, which is messy, and loud. Of course, it's great to show high accuracy for these artificially clean scenarios and datasets, but this isn't our North Star.

At Speechmatics, our North Star is usefulness. We want to create something that works for real product companies looking to provide value in the real world. If we create a product that looks great on paper but fails for our customers, we have failed. If we create a product that only excels in one language or fails to understand a particular demographic and not others, we have failed. Our mission is to Understand Every Voice, and we take that mission seriously.

How then, can you judge the accuracy of a provider? How do you know if it will work with your real-world media?

To do that, you need to get messy.

Making things more realistic  

If the aim is to provide a better indication of how a product will perform in the real world, you intuitively need to make the test data more realistic (we call this representative).

You can do this by throwing in some more varied audio data into the mix.

For example:

• Data from speakers with more varied social and economic backgrounds. 

• Data from speakers with different accents. 

• Data from speakers having a spontaneous ‘natural’ conversation rather than reading from a text. 

• Data from multiple speakers talking to each other to simulate a phone conversation. 

• Data with speakers in noisy environments.

Now what happens when we do this?

Here we look at tests comparing OpenAI Whisper and Speechmatics with different datasets, each designed to address one of the above. Here we're looking at Word Error Rate, to give you a sense of the percentage of mistakes made. In this case, lower is better.

Conversations across broad demographics  

Casual Conversions is Meta's dataset, composed of over 45,000 videos (3,011 participants). The videos feature paid individuals who agreed to participate in the project and explicitly provided age and gender labels themselves. The videos were recorded in the US, with a diverse set of adults in various age, gender and apparent skin tone groups.

(Don't forget, lower is better when it comes to WER.)

This shows that with this data, Speechmatics is making 18.95% fewer errors for those with light skin tones and 23.99% fewer errors for those with dark skin tones.  

Telephone conversations 

Switchboard is a collection of about 2,400 two-sided telephone conversations among 543 speakers (302 male, 241 female) from all areas across the US. It was created to give a close approximation of a telephone call and is therefore close to the kind of audio that Call Centers and CCaaS providers would look to transcribe.

In this scenario, Speechmatics leads again, recording 47.25% fewer errors.