Speechmatics vs Whisper On-Device: how Adobe Premiere's speech engine got rebuilt

In Part One, "The Adobe Story", we looked at the challenge of getting STT working to Adobe's high standards on macOS and Windows, and pushing the boundaries of laptop GPU frameworks to support audio inference acceleration.

In this part, we look at the curve-ball thrown by OpenAI's launch of Whisper as an open-weights model that anyone can use "for free".

Starting with a top-tier cloud speech model that is more accurate than Whisper, the challenge this time was compressing the model to run frugally on laptop-class hardware.

What I'll cover in this post:

• Why quantization was the central technique for bringing a cloud-grade speech model onto consumer hardware

• Where the LLM-era quantization schemes transfer to other models like speech, and the parts of the surrounding optimization chain that make the work hard

• The four things we did to ship

• How our STT compares on a Dell XPS 16 and M1 MacBook Pro with the best Whisper apps

First, The numbers

All figures below come from running 20 minutes of clean audio on:

• Dell XPS 16 9640 — Intel Core Ultra 7 155H, integrated Intel Arc GPU, and discrete NVIDIA RTX 4050 GPU

• 2020 M1 Apple MacBook Pro — integrated GPU and Neural Engine

The Whisper apps all use the Large V3 Turbo model.

Windows laptop

Mac laptop

Note there is no GPU memory column as Apple Silicon Macs have unified memory shared between CPU, GPU and ANE.

The throughput numbers are worth reading twice.

ESL on a single CPU thread transcribes clean audio at 3.9 s/s on Windows or 4.7 s/s on macOS, more than enough to handle real-time with no GPU acceleration. With multi-threaded RTX 4050 on Windows, ESL hits 25.3 s/s in just 1.7 GB of total memory (1.3 GB RAM + 0.4 GB GPU). On macOS it reaches 47.2 s/s.

The closest Whisper runtime on Windows is the large-turbo whisper.cpp build on RTX 4050 (22.1 s/s vs ESL's 25.3 s/s), but it uses 3.2 GB of total memory: nearly double ESL's footprint. On macOS the closest Whisper is WhisperKit CLI on GPU/ANE (11.7 s/s vs ESL's 47.2 s/s).

One caveat: Adobe does not enable the full set of speed optimizations available in our library. Premiere has to allow for dozens or hundreds of concurrent tasks on a min-spec machine, so the deployment leaves headroom for the rest of the application. With the full set of optimizations enabled, the library goes faster still.

How Whisper changed the game

In 2022, OpenAI's release of Whisper transformed the STT world. Here was an open-weights model anyone could use, trained on 680,000 hours of multilingual audio, ready to transcribe in 99 languages. Architecturally it was a single end-to-end model that could run almost entirely on GPU.

Following its release, multiple organisations pitched in to improve speed and memory usage, and dozens of products sprouted up to offer more usable integrations for different use cases.

By the time we started the 2025 rebuild of our Adobe Premiere integration, apps such as MacWhisper, EasyWhisperUI and OfflineTranscribe (powered by WhisperKit, whisper.cpp and faster-whisper) had set a new bar for on-device accuracy, speed and footprint.

We had been Adobe's speech-to-text partner since 2021. To keep our partnership going, the 2025 brief was specific: bring the latest generation of our cloud models onto consumer laptops to outperform Whisper on accuracy, and match or beat it on speed, while staying inside Adobe's resource budget so that Premiere has headroom to smoothly run the dozens of other tasks it handles even on minimum-spec hardware.

Why quantization was the right lever

Like Whisper, we have used the transformer model architecture as the foundation of our STT cloud products since 2022. Unlike Whisper, our acoustic foundation model is trained with self-supervision, so we can pretrain with millions of hours of audio-only data that has no transcript. This gives us on the order of 100x data efficiency when training using audio with matching transcripts.

This data scale and efficiency advantage, combined with model size increases allowed by GPU inference, has enabled us to comfortably beat Whisper with our cloud service accuracy. Those are the models we now needed on a laptop.

The GenAI explosion started by ChatGPT has been driving a huge industry-wide investment in optimizing transformers to fit bigger models on the same servers, and to fit existing models on smaller devices. Quantization has become a central technique in that body of work as it helps with both cases.

Applied effectively, it can reduce the memory a model needs by a factor of eight while sacrificing an amount of accuracy that is barely perceptible, with the memory saving often resulting in a speed-up as well. Applied badly, the results are catastrophic: horribly wrong output or even complete inference failure.

For us, it quickly became clear that quantization was the lever that would get a cloud-grade model to fit on consumer hardware at accuracy close to the cloud version.

The real trick, though, was getting everything else in the optimization chain to behave so that quantization actually delivered its full potential.

Where LLM quantization intuition transfers, and where it breaks

We set out to apply the latest generation of quantization schemes developed for LLMs to speech processing. On the face of it, this sounds easy: just put the model through one of the various popular tools and out will pop a perfectly formed gem!

The reality was not so simple. Yes, the weights were easily quantized to make a small model, but wrangling the surrounding optimization chain so as not to break the model is where the real time went.

Three specifics worth calling out.

1. Optimization is a complex and brittle chain, not a single step.

Model optimization, particularly for GPUs and other hardware accelerators, is a chain of steps applied at different points by different components:

• Export from PyTorch to an intermediate portable format (ONNX)

• Generic optimizations in ONNX such as quantization, shape inference and operator fusion

• Hardware-specific optimization steps such as memory layout and kernel choices (CoreML on macOS; DirectML via ONNX Runtime on Windows)

Each step is written to do the best job it can, and the companies involved are motivated to show off good results. In practice, this means ensuring that popular reference models like resnet50 or distilbert work really well in benchmarks.

The problem is that most optimizations work by recognising specific patterns in the model's inference graph in order to replace small groups of operations with a single fused operation that is mathematically equivalent but optimized for the target framework and hardware. Unfortunately, specific patterns may only appear exactly for popular reference models, making the pattern-matching logic brittle and increasing the risk of errors.

This brittleness compounds as the inference graph is rewritten by the various optimization steps. Changes to one step risk confusing later steps and preventing optimizations from being applied, or from being applied correctly.

2. Audio transformers look a bit different — your mileage may vary.

Even seemingly small changes in a model's definition, such as a subtle tweak to the transformer layers to work better with audio, can make the final inference graph look surprisingly different. This is a real problem since some of the final optimization stages happen inside the GPU driver where only a tiny bit of the graph is visible at any one moment.

This risk is not theoretical. We found that some of the most important optimizations would be silently skipped, while others could actively break the model due to bugs in the tooling or simply because of the inherently risky nature of some optimization techniques.

It's worth noting that the experience here is very different to normal programming language compilers, where the optimizer is conservative about preserving semantics unless you explicitly give it permission to relax key assumptions. In ML optimization, it can feel like surprises are intended: popular tools change semantics by design and potentially quite drastically.

PyTorch V2 export using torch.compile is the clearest example: it deliberately only preserves the tensor operations it saw when tracing one reference input, and will discard conditional logic and ancillary calculations on the assumption that the only thing that matters is the sequence of tensor operations.

Using torch.compile is a bit like hiring a moving company and finding out they rebuilt a new house with everything fixed in the exact positions they saw during the tour. Doors glued shut, oven shelves welded in place, whole rooms missing because they were not part of the walk-through. Useful, until it is not!

3. Precision loss is an extra risk dimension on top of all of that.

Quantization works by shrinking the precision and range of the weights that encode the model's knowledge and thus determine its accuracy. In many respects, this is as risky as it sounds: it is literally exponentially increasing the chance of overflow or underflow in individual operations. Either one can cause a complete collapse in accuracy.

Even if that is avoided, precision loss risks corrupting the calculations in subtle ways that might degrade accuracy, but only in certain situations.

Despite these risks, the technique is often workable in practice because many model layers end up with nicely balanced weights (if model training went well), meaning they have a small range which can be captured with fewer bits, enabling significant savings in memory and potentially in compute time.

Tiny errors at each layer injected by precision reduction can be mitigated during training with quantization-aware training techniques. New techniques are being frequently developed that can do a better job, and advanced block data types have been introduced with hardware support to try and keep slightly more precision with equivalent compression.

No wonder: the stakes are high, as quantization with minimal accuracy loss is a potent lever for improving the economics of AI model deployment.

For the AI model developer, this doesn't happen for free. It is essential to:

1. Identify every section of the model where precision and range loss cannot be tolerated, and instruct the quantization tool to leave those alone

2. Consider the cumulative effect across many layers: even where per-layer error is individually survivable, it can push final accuracy past the edge of acceptable

3. Ensure the quantization changes to the inference graph don't trip up the other optimization steps

In our experience, this process is highly specific to the model and framework involved, and takes real time because of the iterations and debugging needed to reach a good place. You need engineers who deeply understand the model's architecture and can dig deep to inspect layers and ops, check the inference graph for correctness and optimality, and run good tests that will spot subtle changes in the distribution of model predictions. You may need to write scripts that systematically compare the unquantized and quantized activations layer-by-layer to spot where errors creep in.

What we did

There were four key decisions we took that are worth exploring here.

1. Go deep on each case.

The state of the art for ML optimization and quantization is constantly evolving as developers and companies scramble to find an edge. There is no single approach that works universally or every time.

In our case, Apple and Windows devices and frameworks are different enough that we needed separate efforts to do each one justice. We spent several months optimizing our models for CoreML in order to efficiently leverage both the GPU and the Neural Engine on Apple devices. WhisperKit had already achieved excellent results by applying insights from Apple's ANE Transformer paper, which we were able to replicate.

It took a similar amount of effort to do the equivalent for Windows PCs, this time relying on DirectML and the ONNX Runtime to support any GPU with appropriate Windows drivers. Both framework choices had been stipulated by Adobe to ensure the best possible compatibility and reliability, based on their years of experience running ML models on millions of consumer devices.

2. LLM-era quantization schemes, adapted per layer.

Early on, we decided to quantize our cloud service models rather than training models with fewer parameters and basic float16 quantization, as that would struggle to achieve similar accuracy even using distillation to transfer knowledge from our biggest models.

We investigated the latest quantization schemes developed for on-device LLMs like Llama, Gemma, Phi, Qwen and GPT-OSS as the starting point. There is a bewildering array of tools offering different techniques for quantization (often specific to particular inference frameworks), making it tricky to figure out which one was most appropriate for our models.

In the end, we used 6-bit palettization in CoreML for macOS, while on Windows we used INT4 weight-only quantization in DirectML/ORT for decent compression with good hardware support.

3. Isolate precision-sensitive sections and keep them out.

The essential step in quantization is finding and isolating the sections of the model where a loss of precision or range cannot be tolerated, and telling the quantization tool not to modify those layers.

Even where the reduction of precision and range can be tolerated for an individual group of operations, the cumulative effect can unacceptably degrade the accuracy of the model's final output. The work is very model-specific and can be quite intensive and time-consuming to get to good compression with minimal accuracy degradation.

4. Custom export and optimization scripts.

To deal with the various quirks, bugs and limitations of the multi-vendor toolchains, we built our own export and optimization scripts to ensure that specific fused operations were always used in the final optimized graph, without leaving it to the mercy of the pattern-recognition process across the various optimization phases in different components.

These directed optimization steps were crucial for delivering models that reliably achieved the desired accuracy, speed and memory use on the diverse range of devices which matters to Adobe.

We accepted one trade-off explicitly: some very old, low-powered GPUs do not have the hardware support needed to run quantized models efficiently, so on those devices the new model is slower than the previous one, though still much more accurate.

In short, there was no way around it: we needed to master each tech stack at pretty much all levels and steps of the process to get these results.