The Surprising Effects of Emotion in Speech-to-Text Technology

At Speechmatics, we're constantly working to grow our Autonomous Speech Recognition (ASR) capabilities. We've mastered 33 languages, but the research doesn't stop there. To understand language in a more comprehensive, authentic way, we must look at the role of emotion in speech-to-text.

Speech is a naturally rich tapestry of ever-changing emotions. It's a crucial part of human communication, so the boundaries of emotion in speech-to-text will always push. Naturally, we tested our ASR on the beloved TV show Friends' character's voices.

Set in 90's New York, the iconic sitcom follows Ross, Rachel, Monica, Chandler, Phoebe, and Joey through the trials and tribulations of adulthood. The show's rich storytelling ensures a vast array of emotions throughout the seasons, making it the perfect subject for our test.

To analyze the accuracy of our ASR, we used the MELD dataset, which includes 1400 dialogues and 13000 utterances from the show, accompanied by emotion and sentiment annotations. We calculated transcription accuracy for each utterance using Word Error Rate. We then measured ASR's accuracy levels for each character's emotions: fear, neutral, anger, disgust, joy, positive, and negative surprise.

Fear vs Speech Recognition Technology

Amongst the different emotions, fear stands out the most. Compared to 85% neutral accuracy, fear registered an average of 78% - a 7% absolute difference. 

Even if accuracy varies across the other emotional states, none of these differences is statistically significant. You can see in the figure above that the error bars are overlapping, which means that these minor differences are likely due to chance, and in reality, transcription accuracy for neutral and anger, for example, are comparable. But if we take a closer look, we can see the pattern of results changes.   

Monica and Rachel's accuracy drops to 67% and 65% for fear, but not for the rest. Phoebe, Ross, Chandler, and Joey all have a fear accuracy comparable to neutral accuracy. Again, even if these slightly vary, none of the differences is statistically significant – it's probably just chance. But Rachel and Monica's fear accuracy is statistically different to neutral accuracy.

The Difference Between Positive and Negative Surprise

Phoebe, played by Lisa Kudrow, has an emotionally rich past. She's experienced just about every emotion on the spectrum, so it's no surprise that her test results provide the best insight into the impact of emotion in speech-to-text technology.

Whereas Phoebe's negative surprised tone was 93% accurate, positive surprise only recorded a 70% accuracy rating. However, this issue doesn't spread beyond Phoebe, as every other character scored similarly high accuracy levels for both.

Interestingly, Joey's positive surprise accuracy levels were the same as Phoebe's negative surprise – 93%. It's the same emotion, but the character's sentiment drastically impacts the way they speak – as indirectly shown by the difference in accuracy levels. While more work is needed to uncover the true reason as to why that is, highlighting the effect of emotion in speech-to-text helps us better understand the true capabilities of ASR and language in general.

Phoebe's positive surprise and Monica & Rachel's fear are what ASR struggles with the most. It begs the question: why does ASR struggle with these emotional tones? Is it because people's voices reach very high pitch ranges in these emotional states?

One thing is clear. We all have different, unique voices, but they also change constantly. And, amongst many factors, emotions greatly determine these changes. But our voices change in other ways too. Monica and Rachel's fearful speech is similarly unique to Phoebe's when positively surprised.

How Can Emotion Improve Speech-to-Text?

Emotions are as complex as our voices - they are a unique array of variations that characterize our lives daily. ASR must adapt to the different emotions and the way we express all the individual differences in our voices.

If our technology only focused on neutral speech, we would be missing out on many variations of everyday life. We're looking to incorporate as much diversity into our systems as possible with continued research. That doesn't just include different languages and dialects. People are happy, stressed, excited, frightened, and angry, or perhaps everything all at once.

Emotions change the way you speak; we need to change how we listen.

Benedetta Cevoli - Data Science Engineer, Speechmatics