How Does Speaker Diarization Work? A Clear Explainer

Speaker diarization can feel like magic: feed in a recording of several people, get back a transcript that knows who said what. It isn't magic — it's four understandable steps. This explainer walks through how speaker diarization works without drowning you in math, then covers why it sometimes gets things wrong and how to run it privately on your own device.

If you're new to the concept, start with what is speaker diarization; this piece goes under the hood.

The goal, restated

Diarization answers "who spoke when?" Its job is to take a continuous audio stream and carve it into labeled speaker turns — Speaker 1 here, Speaker 2 there — so those labels can be attached to a transcript. Note that diarization on its own doesn't produce words; that's transcription. Diarization produces the turns, which then get lined up with the words.

The four steps

Almost every diarization system, from cloud APIs to open-source pipelines, does these four things in order.

Step 1 — Voice activity detection (find the speech)

First the system separates speech from non-speech. Recordings are full of silence, breaths, keyboard clacks, traffic, and music. Voice activity detection (VAD) marks which slices of the timeline actually contain someone talking, so the rest can be ignored. Everything downstream only looks at the speech regions.

Good VAD matters: if it mistakes noise for speech, you'll get phantom speakers; if it misses quiet talking, you'll lose turns.

Step 2 — Segmentation (cut into chunks)

The speech regions are then sliced into short segments — typically a second or two — at points where the speaker is likely to have changed. Early systems used fixed windows; modern ones detect probable change points. The aim is that each segment contains, as much as possible, only one speaker's voice.

This is also where the hardest problem lives: overlapping speech. When two people talk at once, a segment genuinely contains two voices, and no clean cut exists. Advanced systems try to handle overlap explicitly, but it remains the main source of errors.

Step 3 — Speaker embeddings (fingerprint each voice)

Now the interesting part. Each segment is converted into a speaker embedding — a list of numbers that acts like a fingerprint of that voice. The embedding captures characteristics of how the person sounds (pitch, timbre, vocal tract shape) rather than what they said.

The crucial property: two segments from the same speaker produce embeddings that are numerically close together, while segments from different speakers produce embeddings that are far apart. The words don't matter — the voice does. This is what lets the system tell voices apart without knowing anyone's identity (which is why diarization is different from speaker recognition).

Step 4 — Clustering (group the fingerprints)

Finally, the embeddings are clustered: grouped so that similar ones land together. Each resulting cluster is treated as one speaker. All the segments in cluster A become "Speaker 1," cluster B becomes "Speaker 2," and so on.

Sometimes the system knows the number of speakers ahead of time; often it has to estimate it from the data — which is itself a hard sub-problem. Guess too few and it merges two people; guess too many and it splits one person into two.

Stitching it to the transcript

After clustering, you have a timeline of speaker turns: Speaker 1 from 0:00–0:12, Speaker 2 from 0:12–0:20, and so on. Separately, a transcription engine has produced words with timestamps. The last step aligns the two: each word's timestamp is matched to whichever speaker turn it falls inside, so every word inherits a speaker label.

The result is the thing you actually wanted:

Speaker 1: So should we push the launch to the 15th? Speaker 2: That works, but marketing needs the assets by the 8th.

This is why word-level timestamps matter so much: without them, you can't accurately attach words to the right speaker turn.

Why diarization makes mistakes

Understanding the steps makes the failure modes obvious:

• Overlapping speech (Step 2) — two voices in one segment can't be cleanly assigned. Talking over each other is the number-one killer of accuracy.
• Similar voices (Step 3) — if two people have alike pitch and accent, their embeddings sit close together and clustering merges them.
• Too many speakers (Step 4) — the more people, the harder it is to estimate the count and keep clusters clean. Two is easy; ten is hard.
• Noise and distance (Steps 1 & 3) — background noise and a far-away mic muddy the audio, degrading both detection and embeddings.

Even the best tools slip in large, messy meetings. The fix is almost entirely on the recording side.

How to get more accurate diarization

You have far more control over accuracy than the algorithm does — through how you capture the audio:

• Get the mic close to the speakers. Proximity beats everything. See recording two people or a room.
• Record in a quiet space. Kill background noise before it starts — how to reduce background noise.
• Encourage one voice at a time. A little social nudging beats any overlap-handling model.
• Use decent input. A better phone mic setup or an external mic sharpens the embeddings.

Clean audio in, clean speaker labels out. It's the highest-leverage thing you can do.

Real-time vs after-the-fact diarization

Diarization runs in two modes, and they involve a genuine trade-off:

• Real-time (streaming). Labels appear live as people speak — what you see in a meeting-notes app mid-call. It's convenient but harder: the system must decide "who's talking" without hearing what comes next, so it can revise labels or slip more often.
• Batch (after the recording). The system sees the whole recording before labeling, so it can cluster voices with full context. This generally yields more accurate speaker separation.

For most personal and professional recording, batch diarization on a finished recording is both more accurate and perfectly sufficient — you don't need labels during the conversation, you need a clean transcript after it. That's the mode an on-device recorder uses: capture first, then transcribe and diarize the completed recording.

How many speakers can it handle?

A practical question with a practical answer. Diarization systems either take the speaker count as an input or estimate it from the audio:

• 2 speakers — trivial and highly accurate.
• 3–6 — reliable on clean audio, with occasional merges of similar voices.
• 7+ — accuracy falls off; miscounting and merging become common.

If you know the number of speakers ahead of time and can tell the tool, you remove one source of error (the count estimate). Either way, the fewer and more distinct the voices, the better the result — which is why a 1:1 interview diarizes almost perfectly while a ten-person focus group always needs a correction pass.

Diarization vs the transcript timeline

One subtle point that trips people up: diarization works on the audio timeline (segments of sound), while transcription works on words. The magic is in the alignment step, which needs accurate word-level timestamps to attach each word to the right speaker turn. If timestamps are coarse, a word spoken right at a speaker change can be assigned to the wrong person. This is why tools that produce precise word-level timing (rather than just sentence-level) tend to give cleaner speaker labels — the boundaries line up correctly.

Where diarization runs — and why it matters

The four steps can run in three places, with very different privacy implications:

• Cloud services (Otter, Fireflies, Rev; APIs like AssemblyAI, Deepgram, Google, AWS) run them on their servers — which means uploading your audio.
• Open-source on your computer (Whisper + pyannote / WhisperX) runs locally but typically needs a HuggingFace account, command-line setup, and often a GPU — see a no-code WhisperX alternative.
• On-device apps run the pipeline efficiently on your phone, with nothing uploaded.

That last option is increasingly practical. BlackBox performs transcription and speaker labeling on-device, so all four steps happen on your phone. Your audio — often your most sensitive conversations — never leaves the device, there's no account, and it works offline on iPhone and Android. You get the mechanism described above without the cloud exposure.

A mental model you can keep

If you forget the jargon, remember this analogy. Imagine sorting a shuffled stack of photos of people you've never met into piles, one pile per person. You don't know anyone's name — you just group faces that look alike. Diarization does the same with voices: it turns each snippet of speech into a "face" (the embedding), then sorts them into piles (the clusters), one per speaker. It never learns who anyone is; it only groups by similarity. Then it writes "Speaker 1, Speaker 2" on the piles, and you rename them to the real people afterward.

That analogy also explains the failure modes at a glance: two people who sound alike are like two people who look alike — hard to sort into separate piles. And when two people talk at once, it's like a photo with two faces in it — which pile does it go in? Those are exactly the cases where diarization struggles, and why clean, one-at-a-time audio sorts so much better.

The bottom line

Speaker diarization works in four steps: detect speech, segment it, fingerprint each voice as an embedding, and cluster those fingerprints into speakers — then align the result with a timestamped transcript so every word gets a speaker label. Its mistakes come from overlap, similar voices, crowds, and noise, all of which you fix by recording clean audio. And because the whole pipeline can run on-device, you don't have to trade privacy to get it. BlackBox does diarization on your phone — private, offline, and free on iOS and Android.