What Is Word Error Rate (WER) — and Why Vendor Numbers Mislead

How is word error rate calculated?

WER comes from aligning the ASR hypothesis against a human reference transcript using minimum edit distance (the Levenshtein alignment), then counting three error types: substitutions (the model wrote the wrong word), deletions (it dropped a word), and insertions (it hallucinated one). The formula is WER = (S + D + I) / N, where N is the word count of the reference. A 12% WER on a 100-word call means roughly 12 word-level mistakes.

Two properties surprise people. First, WER can exceed 100% — a model that hallucinates long repeated phrases on silence or hold music will rack up insertions faster than the reference has words. Whisper-family models are known to do exactly this on long non-speech segments. Second, WER is unforgiving of trivial differences: "five" vs "5", or two valid Arabic spellings of the same word, count as full substitutions unless the scoring pipeline normalizes them away. That single choice — what the normalizer collapses — is where most benchmark games are played.

The metric has real history behind it. NIST has run formal ASR evaluations since the late 1980s — the Hub4 broadcast-news and Hub5 conversational-telephone benchmarks, then the Rich Transcription series — and the field standardized on WER scored by NIST's sclite tooling precisely so that labs could not each invent their own flattering arithmetic. Vendor marketing pages today rarely inherit that discipline.

Why do vendor WER numbers mislead?

Three mechanisms account for almost every too-good-to-be-true number you will see on an ASR pricing page.

• Test-set selection. Scoring on clean, read-speech corpora (LibriSpeech-style audio, or Common Voice prompts read into a good microphone) produces single-digit WER that has no relationship to an 8 kHz call-center recording with cross-talk and hold-music bleed. If the vendor was also trained on the public test set — common with scraped corpora — the number measures memorization, not accuracy.
• Normalization games. The Whisper paper (Radford et al., 2022) ships an aggressive English text normalizer, and reported numbers change materially depending on whether it is applied. For Arabic the effect is larger: alef variants (أ/إ/آ/ا), taa marbuta vs haa, and hamza seats each create spelling pairs that one normalizer merges and another counts as errors. Two teams can score the same hypothesis against the same reference and land 5+ WER points apart.
• Dialect mismatch. A model evaluated on Modern Standard Arabic broadcast news says nothing about Khaleeji or Levantine call audio. In our own evaluation, the same engine that scores 6-9% WER on clean MSA scores 8-12% on clear Khaleeji calls — and third-party APIs that look competitive on MSA fall to 16-24% on the dialectal audio. "Arabic WER: 9%" is not a lie; it is just an answer to a question you did not ask.

What is a good WER for call center transcription?

For real telephony audio, calibrate expectations by condition rather than chasing a single number. On clear Gulf Arabic calls (SNR above 15 dB) a well-tuned Whisper large-v3-turbo pipeline lands in the 8-12% WER range; Levantine runs a few points higher; noisy or heavily code-switched audio higher still. CallScribe publishes exactly this kind of per-dialect, per-condition breakdown — with the dataset composition and known limitations — on its model card, because a range with stated conditions is verifiable and a bare number is not.

WER also does not weight words by importance. A 10% WER transcript that nails every name, amount, and account number is more useful than an 8% transcript that garbles them, which is why teams scoring compliance calls often track entity-level accuracy alongside WER. Treat WER as a comparative instrument — same audio, same reference, same normalizer, different engines — not as an absolute grade.

How should you evaluate an ASR vendor's real WER?

Run the benchmark yourself, on your own calls. Pull 20-50 representative recordings — your dialect mix, your telephony stack, your noise floor — and have a native speaker produce reference transcripts. Fix one normalization scheme in writing before scoring anything, score every candidate engine with the same tooling, and segment results by dialect and audio quality instead of averaging everything into one figure. The whole exercise costs a few days and will contradict at least one marketing page.

Be suspicious of any vendor who will not let you run this test. Self-serve trials exist precisely so the claim can be checked: upload the same ten calls to each candidate and count the damage. Published numbers earn trust only when the methodology — test set, normalizer, conditions — is published with them.