Benchmarking Arabic ASR: What Actually Works

Every vendor with an Arabic speech-to-text API will tell you their model is state of the art. The marketing page shows an impressive WER number — maybe single digits, maybe mid-teens — and a generic mention of "dialect support". Then you feed the API a real Khaleeji call center recording and the output is nonsense. Why does this keep happening, and what does a fair Arabic ASR benchmark actually look like? This post walks through the models we evaluated while building CallScribe, the datasets we tested them on, and the evaluation mistakes that make most public comparisons useless.

The Model Landscape in 2026

There are roughly four families of multilingual speech models worth comparing on Arabic today: Whisper large-v3 and its distilled turbo variant from OpenAI, wav2vec2-XLS-R and its Arabic fine-tunes from Meta, SeamlessM4T (also from Meta), and the commercial APIs from AWS Transcribe and Google Cloud Speech-to-Text. Open-source Arabic-only models like Elyadata/wav2vec2-large-xlsr-53-arabic and various university research checkpoints round out the landscape.

Whisper's advantage is its massive pretraining corpus and its robustness to noisy, real-world audio. It was trained on roughly 680,000 hours of weakly supervised multilingual audio, including a meaningful chunk of Arabic. The large-v3-turbo variant shipped by OpenAI in 2024 keeps most of the accuracy of large-v3 at roughly six times the inference speed — an enormous quality-of-life win for anyone running it in production. For our workload (call center recordings, 2-5 minute files, mostly Gulf Arabic) Whisper large-v3-turbo consistently outperformed every open-source alternative we tested.

wav2vec2-XLS-R is a different design philosophy: self-supervised pretraining on raw audio followed by supervised fine-tuning on a labeled dataset. The Arabic fine-tunes available on Hugging Face are useful but limited — most were trained on MGB-2 (broadcast news) or Common Voice Arabic, which does not match call center audio at all. They work fine on clean MSA and fall apart on Khaleeji. We tried three of the most popular ones and they all underperformed Whisper large-v3-turbo by 8-15 WER points.

Per-Dialect Results

Our internal benchmark ran on a 200-recording corpus of GCC call center audio, hand-annotated with reference transcripts and speaker labels. The table below summarizes word error rates by dialect on clear audio (SNR above 15 dB). All numbers are from our own evaluation — they should not be compared to published benchmarks on different datasets without caveats.

• Khaleeji (clear audio): Whisper large-v3-turbo 8-12%, wav2vec2-XLS-R Arabic fine-tunes 18-26%, AWS Transcribe 16-22%, Google STT 17-24%.
• Levantine (clear audio): Whisper large-v3-turbo 12-16%, wav2vec2 fine-tunes 22-30%, AWS Transcribe 19-27%, Google STT 21-29%.
• Egyptian (clear audio): Whisper large-v3-turbo 10-14%, wav2vec2 fine-tunes 18-24%, AWS Transcribe 15-21%, Google STT 16-22%.
• MSA news-style (clear audio): Whisper large-v3-turbo 6-9%, wav2vec2 fine-tunes 8-13%, AWS Transcribe 7-11%, Google STT 8-12%.

The headline: Whisper large-v3-turbo wins on every dialect in our corpus, but the margin narrows dramatically on MSA — which is unsurprising, since MSA is what most open datasets teach models to do. The margin widens on noisy audio and dialect-heavy recordings, which is exactly the use case we care about.

Code-Switching Behavior

GCC business calls regularly switch between Arabic and English mid-sentence — sometimes mid-word. "Okay يعني I will check the account و I will call you back" is a normal utterance in a Dubai customer service call. How well do these models handle that?

Whisper handles code-switching cleanly in most cases. The language detector runs per-chunk, and the decoder is capable of emitting both Arabic and English tokens in the same segment. We measured roughly a 2-3 WER-point penalty on code-switched utterances compared to single-language utterances of similar length — meaningful but not catastrophic. wav2vec2 fine-tunes tend to collapse on code-switching because they were trained with a fixed vocabulary and language prior; English tokens come out as Arabic phonetic approximations. AWS Transcribe requires you to pick a language up front, and Google STT has a code-switching mode that works on paper but produced inconsistent results in our testing.

Evaluation Pitfalls

The biggest mistake in public Arabic ASR comparisons is comparing models that were trained on the benchmark dataset against models that were not. If you evaluate Model A on Common Voice Arabic and Model A was trained on Common Voice Arabic, you are measuring overfitting, not generalization. MGB-2, Common Voice, and Tashkeela all show up in multiple training sets. Any serious benchmark must hold out a dataset the models have never seen.

The second mistake is reporting a single WER number. WER depends enormously on audio quality, dialect, and call length. A vendor showing 5% WER on a 30-second MSA news clip is telling you almost nothing about how the model will perform on an 8-minute Khaleeji call with hold music bleed. Segment your evaluation by the conditions your production data actually contains.

The third mistake is ignoring normalization. Arabic text has multiple valid spellings for the same word (alef variants, taa marbuta vs. taa, hamza placement), and WER calculations either collapse these variants or treat them as errors depending on the normalization pipeline. If you do not publish the normalizer, your numbers cannot be reproduced.

What We Ship

CallScribe ships Whisper large-v3-turbo as the primary engine, with pyannote.audio 3.x for diarization and a multilingual sentiment model on top of the transcript. We run all inference on our own GPUs, not a third-party API. The eval harness we built internally is on the roadmap to open source — once the dataset licensing is cleared, we will release a reproducible benchmark so anyone can rerun the numbers in this post on their own audio.

Until then, the best thing you can do if you care about Arabic ASR quality is run a model on your calls, compute WER against your reference transcripts, and ignore every vendor that will not let you do that.