Speaker ID Without Voiceprints
How per-speaker text signatures lifted overall accuracy from the low 40s to 58% (and to 83% for speakers with history), and the ten plausible ideas we killed with data along the way.
In One Paragraph
In a meeting transcript, who said what determines who owns each summary point and action item. We identify the speaker not from their voice, but from the words they have actually used.
The real wall wasn't scarce data, it was teammates too similar to tell apart. The fix was each speaker's text signature (a lexical fingerprint built from their past utterances), which lifted overall accuracy +15pt (to 58%) and to 83% for speakers with history.
We put 18 ideas on the table: 7 shipped, 10 killed, 1 lined up next. Embeddings, a bigger model, personal memory: none helped. Measurement kept the noise from shipping.
01 · The Problem — The day three people became one
We ran speaker identification, and three different speakers were assigned the same identity, each with more than 80% confidence. The meeting host wasn't even on the list.
That single frame captures the whole problem. The model was more confident when it was wrong, and for a long time we were judging its accuracy by feel. This is the story of turning that feel into numbers.
02 · Why It's Hard — Why we skipped the voiceprint shortcut
The easy path is a voiceprint: register each voice and match. We didn't take it. Meetings mostly start impromptu and verbally, with no enrollment and no attendee list. The one signal left is what people actually said.
So we diagnosed why we miss before trying to fix it. And the conventional wisdom didn't hold.
The problem wasn't finding a person. It was telling one person apart from others who are similar.
Teammates talk about the same projects in the same terms. So the bottleneck was discrimination, not coverage, and the model confidently picked the wrong colleague.
A diagnostic on one user's 28 labels: 32% correct, 46% confused with a roster teammate (100% of those confusions were roster members), 21% abstained (no name at all), and a small remainder of self-mistakes. In other words, errors were almost never off-roster garbage; they were a confident mix-up with a teammate, or a give-up. The user with the best coverage was actually the most error-prone.
The signal wasn't missing; people were simply too alike to separate. The direction was set: not more knowledge, but more personal language.
03 · The Signal We Found — What set people apart was their own language
The signal we found was personal language. Teammates were similar in meaning, but different in vocabulary. What separated one person from another was their lexical footprint: not what they talked about, but which words they tended to use. The pipeline flows in three stages on this finding: signature (vocabulary) → stance (how they talk) → anchor gate (sanity constraints). Only when a speaker has no history to fill this signal does structural context (roster, roles: the ontology, later) step in.
① Text signature: a lexical fingerprint instead of a voice
This is the heart of the project. We gather each speaker's confirmed, labeled utterances from past meetings and build a TF-IDF-weighted vector of their distinctive vocabulary. Common words are damped; the project names, jargon, and verbal habits only they use are lifted, a "text fingerprint." We match a new meeting's anonymous utterances to the nearest fingerprint.
The key insight: the old approach failed because it compared everyone against similar role-summary profiles. A signature compares against the language each person has actually used. In other words, it's the "source-based identification" idea we had discussed, implemented properly.
We confirmed the signal is strong before building on it. Untuned, the raw signature alone already reached 73% accuracy on warm speakers, twice the ~37% of the identifier at the time (two later refinements take it to 82%, see the climb in section 05). And it holds up without an attendee list (open-set), so it works with no upfront input, matching our minimize-input direction.
② Stance: not what you say, but how you say it
There's one more axis vocabulary misses: conversational stance. Who directs and asks ("could you...", "how about...?") versus who reports and defers ("done", "understood"). People differ on this even when saying the same thing, and it helps most where history is thin. This feature pushed warm accuracy the rest of the way to 82%.
③ Anchor gate: filtering with common sense
Finally, high-precision textual cues (anchors) reshape the result. A self-introduction is promoted to a confirmed signal; negative cues like being addressed by name or an honorific are rejected; a guess with no supporting cue is downgraded. This is the direct remedy for the overconfidence diagnosed earlier: an evidence-free guess is now downgraded to low confidence. (We also tried recalibrating confidence via the prompt itself; it didn't help and was reverted, see the ledger in section 06.)
It also enforces one person, at most once per meeting, the gate that blocks the opening "three people, one person" failure. But the rule is designed around edge cases, not as a cure-all. Duplicates are removed per speaker ID, not per name, so genuine namesakes survive; a downgraded candidate isn't discarded but handed to the LLM fallback for a second look. Over-segmentation, where one person is split across many segments, is handled separately by a speaker-count cap, not this rule.
04 · Measurement Discipline — Trading feel for a backtest
Before moving any idea to production, we built a backtest harness that replays identification over meetings that already have ground-truth labels. It evaluates a full condition (roster intact, production-faithful) alongside a cold-start condition (the meeting's people removed from the roster). This harness became the judge for every hypothesis that followed.
Scale, up front: the harness measures 5 users and 22–30 meetings, repeated 1–3 times. It's not a large corpus, so read the figures within that scale. Two terms, fixed here: warm = a speaker with prior confirmed-labeled utterances, so a signature exists; cold = a first-time speaker with no such history. The cold-start condition above imitates the cold case.
We fixed one primary metric: the backtest's overall accuracy (the average over all speaker labels, under the production policy). Every precision / warm / percentage-point figure that follows is a supporting slice of that primary metric. When reading a slice, always read what it's a subset of.
The baseline was sobering.
The measurements quickly ruled out three tempting answers. More meetings left accuracy flat (37.9%→37.3%); injecting personal memory added 0.0pt; a bigger model was even more confidently wrong on off-roster people (and blocked by our cost cap anyway). Once those easy answers were ruled out, only one path remained, the one the diagnosis had already pointed to: discrimination.
A rule was set here: an improvement not proven in numbers is not an improvement. From then on, every idea had to clear the backtest before it could ship.
05 · The Accuracy Climb — From raw model to signature
When the direction (discrimination) met the tool (signatures), accuracy jumped. It's the result of switching an LLM-only identifier to signature-first (same eval set, same policy, 3 repetitions).
Lexical weighting strips the filler and backchannels everyone uses, leaving only distinctive vocabulary; conversational stance adds the directs-vs-reports axis, lifting even speakers with limited history. Both cost zero extra API calls, and because every improvement is server-side, web and mobile benefited at once with no app rebuild.
06 · The Hypothesis Ledger — We killed more ideas than we kept
This table is where the story gets honest. We worked through 18 ideas: 7 we shipped, 10 we killed with data, and the last, ontology enrichment, is queued as the next lever. Had we gone by feel, several of those 10 would surely have shipped.
Shipped
One way to read the table: shipping something doesn't make it an accuracy lever. The person-uniqueness fix that killed the opening bug barely moved overall accuracy, flat within measurement noise (the comparison arm swung by the same amount). What it did move: non-self precision from 54.8% to 57.3%, overconfident errors from 13 to 10, and above all it removed the trust-breaking "three speakers, one person" failure. It fixed the honesty of the accuracy, not the accuracy number. That distinction is only visible when you overlay a few supporting slices on one primary metric.
Two killed experiments taught the most. Semantic embeddings betrayed intuition head-on. We assumed embedding sentences by "meaning" would separate people better, but within-team distinction rides on each person's distinctive vocabulary, not general meaning, so smoothing to meaning erased the very individuality (−10 to −15pt, open-warm −11pt). And this time we observed the reason directly. Measuring cross-meeting cosine between different speakers, TF-IDF kept them apart at 0.073 while embeddings pulled them to 0.806: in our data, same-domain teammates were represented far too similarly. As a result the embedding's speaker separation (same − different) was 0.032, narrower than TF-IDF's 0.054. Not a generality, but an observation on our data: TF-IDF's lexical specificity was a strength, not a limitation.
The cold-start self-intro bootstrap was the same. The assumption was that even in a first meeting people introduce themselves, but recurring meetings have no self-intros, and verbal meetings have no attendee list. It triggered on 25% of real notes but produced zero usable assignments, surfacing only role nouns and parse noise. The conclusion: transcript parsing is the wrong instrument for cold-start.
We state the measurement limits too. These figures come from a backtest of 5 users, 22–30 meetings, 1–3 repetitions. Aggregate counts like precision and wrong-applies are stable across runs, but a single-digit change like "13→10" should be read as a direction, not a confirmed signal. If measurement is an asset, noting its variance is part of the argument.
07 · What We Learned — The ten launches measurement prevented
Every killed idea was "plausible." Embeddings, a bigger model, personal memory, an ensemble. By feel we'd have shipped at least a few, and accuracy would have quietly dropped. Without measurement, we'd have shipped our own noise.
And the real lever wasn't a shiny new technique, it was the fact the diagnosis pointed to first: discrimination within the team, not coverage. Once we dropped the voice and used each person's language as the signal, the low-40s raw LLM became 58% (83% warm).
It's not over. Two axes remain. One is first-time speakers (cold-start, the ledger's next lever): a signature needs history to work. In a first meeting with no history, ontology fills the gap: org chart, roles, and domain knowledge impose role-based constraints ("this sounds like the FE lead", "only the team lead makes this call"), narrowing candidates even when the signature is silent. The other is confidence calibration: after the anchor gate, the actual accuracy of non-self predictions offered at ≥0.8 confidence rose from 14% to 75%. Honesty is restored, but tuning confidence more precisely is still open (prompt recalibration failed, so the next attempt needs a different approach).
The lesson of this case is one line: don't guess, measure. And a measurement harness is not a feature, it's an asset: build it once and every later hypothesis is judged on the same scale. In hindsight, the most important thing we built wasn't the speaker identifier. It was the measurement harness that let us keep getting speaker ID wrong until we could tell which ideas were real signal.
Comments