How to Benchmark ASR Vendors on Your Own Calls

Why vendor WER claims don't transfer to your call center

Every ASR vendor publishes a word error rate, and almost none of those numbers will match what you see on your own queue. Published WER is usually measured on clean, consented benchmark corpora — often English-heavy, often read speech or well-mic'd meetings. Your reality is 8 kHz telephony codecs, hold music bleeding into speech, agents in Cairo talking to customers in Riyadh, and Arabic-English code-switching mid-sentence. CallScribe publishes 85-95% word-level accuracy on clear audio above 15 dB SNR, and even we will tell you: validate that range on your own recordings before you sign anything.

The fix is a structured evaluation — a bake-off — where every shortlisted vendor transcribes the same sample of your real calls and you score the outputs yourself against human-made reference transcripts. It takes roughly two weeks of part-time effort and removes nearly all of the guesswork from the purchase.

Step 1: Build a representative sample

Your sample must mirror production traffic, not flatter it. Thirty to fifty calls totaling 3-5 hours is enough for a defensible comparison. Stratify deliberately across the dimensions that actually move accuracy.

Dialect mix matters most for Arabic operations. If your traffic is 60% Khaleeji, 25% Egyptian, and 15% Levantine, your sample should be too — a vendor that excels on Modern Standard Arabic broadcast audio can collapse on Gulf dialect telephone speech. Pull the SNR spread from your recording platform as well: include your worst quartile of audio quality, not just clean calls, because that is where vendors separate. Finally, keep the original codec. Transcoding GSM or G.711 telephony audio to clean WAV before testing hides exactly the degradation you are trying to measure.

• 30-50 calls, 3-5 hours total audio, pulled from the last 90 days
• Dialect proportions matching your real queue (e.g., Khaleeji / Egyptian / Levantine / MSA)
• SNR spread: include your noisiest quartile, not just clear calls
• Original telephony codecs preserved (G.711, GSM, Opus) — no re-encoding
• A mix of call lengths, including long calls with hold segments
• At least a few calls with heavy Arabic-English code-switching

Step 2: Create reference transcripts

References are the expensive part and the part teams most often skip. Have a fluent speaker of each dialect transcribe the sample verbatim — including fillers, repairs, and code-switched English — then have a second person review a 20% spot-check. Write a short normalization guide first: how you will handle hamza variants, taa marbuta, numerals, and English brand names, because inconsistent reference conventions inflate every vendor's error rate equally and noisily. Budget roughly 4-6 hours of transcription effort per hour of audio.

Step 3: Compute WER and CER with open tools

Word error rate is (substitutions + deletions + insertions) divided by reference word count. Do not compute it by hand — use jiwer, the open-source Python library that is the de facto standard for this. A ten-line script loads each vendor's output and your reference, applies the same normalization to both, and reports WER, CER, and the error breakdown.

For Arabic specifically, report character error rate alongside WER. Arabic's rich morphology means a single wrong clitic flips an entire word to an error even when the transcript is perfectly usable, so CER often reflects practical readability better. Normalize both sides identically — strip diacritics, unify alef and yaa variants — and document the normalization so the comparison is reproducible. CallScribe's own /model-card page documents this methodology per dialect; ask every vendor for the equivalent.

Step 4: Score diarization and sentiment, not just words

If you are buying for QA, who-said-what matters as much as the words. Score diarization error rate (DER) using pyannote.metrics against a reference speaker map — for two-party telephone calls a usable system should attribute the large majority of speech time to the correct side. CallScribe runs pyannote.audio for diarization on every call; vendors that charge diarization as an add-on (Deepgram prices it at roughly $0.0015/min on top of base) should be evaluated with the add-on enabled, since that is the configuration you would buy.

Sentiment is harder to score formally, so spot-check it: pick ten calls where you know the outcome (an escalation, a save, a complaint), and check whether each vendor's per-segment sentiment trajectory matches what a supervisor hears. You are looking for directional agreement, not perfection.

Step 5: Model cost at your real volume

Accuracy without cost modeling is half an evaluation. Take your actual monthly minutes and price every vendor at that number, including add-ons. At 500 minutes per month: Whisper API is about $3 (raw transcript only, no diarization), Deepgram Nova-2 with diarization is roughly $2.90, AssemblyAI is about $3.10, Sonix is $50 at $0.10/min, Rev AI is $125 at $0.25/min, and CallScribe Business is $29 flat including diarization, sentiment, and audio quality scoring. The per-minute APIs are dramatically cheaper at low volume — but they deliver raw transcripts, so add the engineering cost of building diarization, sentiment, and QA reporting yourself before comparing totals.

The 2-week evaluation plan

Here is the full bake-off as a schedule a QA lead can actually run alongside the day job.

• Days 1-2: Pull and stratify the 30-50 call sample; get consent/compliance sign-off for sharing audio with vendors
• Days 3-6: Create reference transcripts (with the dialect-aware normalization guide written first)
• Day 7: Second-pass review of 20% of references; freeze the reference set
• Days 8-9: Run the same audio through every shortlisted vendor — trial tiers cover this (CallScribe Free/Business trial, Deepgram's $200 credits, AssemblyAI's $50 credits)
• Day 10: Score WER/CER with jiwer and DER with pyannote.metrics; tabulate per dialect and per SNR band
• Days 11-12: Sentiment spot-checks on 10 known-outcome calls; usability review of exports and dashboards with the QA team
• Days 13-14: Cost model at real volume, weighted decision matrix, write-up for the buying committee