Speaker Diarization Explained: How ASR Knows Who Said What

What does speaker diarization actually do?

Transcription answers "what was said"; diarization answers "who said it, and when". The output is a set of time-stamped segments, each tagged with an anonymous speaker label — SPEAKER_00, SPEAKER_01 — that downstream logic maps to roles like agent and customer. Diarization is unsupervised: the system does not know the speakers' identities and does not need enrollment samples. That distinguishes it from speaker identification (matching a voice to a known person) and speaker verification (confirming a claimed identity).

For call-center analytics, diarization is not optional garnish. Talk-listen ratio, interruption counts, per-speaker sentiment, and "did the agent read the disclosure" checks are all meaningless unless every utterance is attributed to the right side of the call. A transcript with perfect words and shuffled speakers fails QA review just as badly as a transcript with bad words.

How does a modern diarization pipeline work?

The dominant open-source architecture — pyannote.audio, developed by Hervé Bredin and collaborators — chains three stages, and most commercial systems follow the same recipe.

Errors compound across the stages: a sloppy VAD feeds garbage segments to the embedding model, and weak embeddings make the clusterer merge speakers. That is why diarization quality varies far more across vendors than raw ASR quality does — there are three models to get right, not one.

• Voice activity detection (VAD) and segmentation. A neural model marks where speech exists and where speaker turns change, discarding silence, hold music, and noise. pyannote's segmentation model also flags overlapped speech — two people talking at once — which legacy systems simply mislabeled.
• Speaker embeddings. Each speech segment is mapped to a fixed-length vector (x-vector or ECAPA-TDNN style) trained so that the same voice lands close together in the embedding space and different voices land far apart — across phones, languages, and noise conditions.
• Clustering. The embedding vectors are grouped — agglomerative hierarchical clustering or spectral clustering, with VBx-style refinement — and the number of clusters becomes the number of detected speakers. Knowing the speaker count in advance (two, for most service calls) measurably improves results, which is why good APIs let you pass it as a hint.

What is diarization error rate (DER)?

DER is to diarization what WER is to transcription: the fraction of audio time that is attributed to the wrong speaker. It sums three components — missed speech (someone spoke, the system marked silence), false alarm (the system marked speech where there was none), and speaker confusion (speech assigned to the wrong cluster). The DIHARD challenges (2018-2020), organized by Neville Ryant and colleagues, became the field's stress test by scoring DER with no forgiveness collar on deliberately hard audio — and demonstrated that systems scoring well on clean meetings degrade sharply on conversational telephone speech.

Calibrate expectations accordingly: single digits DER on clean two-speaker audio is achievable; real telephony with overlap, speakerphones, and background voices commonly runs 10-20%. As with WER, a vendor quoting one DER number without naming the test conditions is quoting marketing, not measurement.

Why does mono vs stereo call audio matter so much?

The single highest-leverage decision happens before any model runs: how your telephony platform records the call. Stereo (dual-channel) recordings put the agent on one channel and the customer on the other, so speaker attribution becomes a solved problem — each channel is transcribed separately and diarization is only needed for rare multi-party segments. Mono recordings mix both voices into one waveform and force the full embedding-and-clustering pipeline to untangle them, overlaps and all.

If you control your recording stack — Asterisk, Twilio, Genesys, Avaya — enabling dual-channel recording is the cheapest diarization upgrade available. Where mono is unavoidable (WhatsApp voice notes, ad-hoc meeting recordings, legacy archives), pipeline quality is decisive. CallScribe runs pyannote.audio 3.x diarization on its own GPUs and attributes per-speaker sentiment on top, so a mono Khaleeji service call still comes back as a structured agent-customer dialogue rather than an unattributed text blob.

One more telephony trap: 8 kHz narrowband audio does not just hurt word accuracy — it strips the high-frequency speaker cues that embeddings rely on, so two same-gender speakers on a compressed line are genuinely harder to separate. Diarization quality and audio quality are the same battle.