feat(diarize): per-channel acoustic speaker diarization (Sortformer) - #455
Conversation
Optic00
left a comment
Optic00
left a comment
There was a problem hiding this comment.
I ran this branch's sidecar against an independent diarization benchmark rather than reading
the diff alone. Measured on one M5 Air, scored with dscore at a 0.25 s collar against the AMI
reference RTTMs from BUTSpeechFIT/AMI-diarization-setup, on the test split (16 meetings,
9.06 h, Array1-01, one distant room microphone) - the closest public stand-in for what this
PR targets, several people sharing one channel.
It works, and the margin is large
| system | DER | JER |
|---|---|---|
| all remote speech under one label (today's system-audio channel) | 71.09 | 89.82 |
| this branch's sidecar | 29.24 | 39.46 |
Throughput ~105x realtime, about 35 s for a one-hour meeting. This is strong evidence that
the PR materially improves diarization on the case it targets. (FluidAudio's own CLI scored
33.55 on the same material, but that run used a different library version and a likely
different code path, so I would not read anything into the gap either way.)
To be precise about the baseline: it is a one-label system scored on single-channel audio,
so it stands in for the channel where Steno is blind today, not for the whole app - the mic
channel already gets "You" right by construction.
Two claims in the code comments I could not reproduce
The .highContextV2 switch made no measurable difference here. I rebuilt the sidecar with
sortformerHighContextMinDuration raised so .default is always chosen, and reran all 16
meetings: DER identical to two decimals, wall-clock identical too.
AMI meetings are 14-40 minutes, so this does not contradict the PR body's "~6.6x faster" if
that was measured on the multi-hour recordings you tested against. If the gain only appears
past some length, that would explain why I could not reproduce it on shorter material. Not
asking for a change.
Forcing .cpuAndNeuralEngine costs a little more than the comment implies. With
STENOAI_DIARIZE_COMPUTE_UNITS=all: DER 28.89 (vs 29.24) and RTFx 145 (vs 105), so GPU is
marginally more accurate and ~1.4x faster. For the live-recording path the power-first
default still looks right, and the env-var escape hatch is already there. Just recording the
numbers, since the comment reasons about the trade without them.
The phantom-speaker case, which came out better than I expected
EN2002c has 3 reference speakers and both this sidecar and upstream report 4 - the model
fills its slots whether the people exist or not. I also built a 2-speaker fixture (28.8 s,
two macOS say voices, distinctness verified spectrally first) where 0.48 s at the first
speaker change goes to a third cluster holding 1.7 % of speech.
CHANNEL_DOMINANCE_THRESHOLD = 0.92 does not catch it, since the dominant cluster there
holds only 57 %.
It does not reach the user. I drove transcribe_diarised on that file through the real
Parakeet path: the output carries exactly [You] and [Speaker 2], both correct, and no
Speaker 3. The phantom overlaps only a 1.44 s ASR segment, stays under
LONG_SENTENCE_SPLIT_THRESHOLD_S, is assigned whole to the nearest real cluster, then
dropped for carrying no text. So the dominance gate is not the only thing protecting you, and
the design holds under a case built specifically to break it. I had drafted a suggestion for
an extra minimum-cluster guard and dropped it - it would buy nothing observable and would
misattribute the participant who says "approved" once in an hour.
One unrelated thing visible in that same run: a 5.92 s sentence crossing a real speaker
boundary triggers word-level splitting, and the trailing full stop lands past the boundary,
so it opens the next turn ([Speaker 2] ... around the hiring plan / [You] . Sure, let me pull that up.). Cosmetic, and only for sentences past the 5 s threshold that straddle a
turn change.
Scope: shipping diarization before identity looks right to me
Today a three-person call renders every remote utterance as one undifferentiated Others.
Anonymous Speaker 1/2/3 is the normal first rung, turn structure is what makes such a
transcript readable, and the fallback means any failure lands exactly where users are now. I
do not think this needs the identity PR to be worth merging.
One caveat worth stating now rather than after: anonymous labels pay off in the transcript.
In summaries, search and action items, "Speaker 2 owns the follow-up" has no stable referent.
That is an argument for not letting the identity PR sit, not against merging this one.
Related: the summariser now carries speaker attributions it did not have before, and a
confidently wrong attribution is a failure class Others could not produce. Nothing I
measured suggests it is likely, but it is what I would watch first in front of users.
Since the cap will shape that follow-up, I put the measurements in #359 rather than here
- you flagged the 4-speaker limit there yourself, and the short version is that a supplied
participant count is worth real DER, which makes it a cheap field on a form that is going to
ask about speakers anyway. Nothing in it argues against this PR's backend choice; if
anything it vindicates it for a pipeline that has to estimate.
Caveats
No German, and no non-English validation at all - my German fixtures turned out to be
degenerate (one synthetic voice recorded four times), so they measure nothing. AMI is
meeting-room audio, not VoIP system audio. I would not extrapolate 29.24 to either.
Verified in passing
The stdout contamination handling is right, and worth saying because it looks like
over-engineering until you hit it: my run produced exactly the E5RT encountered an STL exception. msg = unordered_map::at: key not found. line ahead of the JSON, with clean
stderr. Scanning every [ with a real decoder and keeping the last segment-shaped array is
what makes that survivable.
Also: keep the Package.resolved pin. The newer FluidAudio I compared against scored worse
on this material, so whoever lifts it should measure first.
Happy to share the run tables, the control runs, or the harness. Nice piece of work - the
fallback discipline in particular is what makes this reviewable as a single change.
|
One concrete follow-up to my review above, which I would rather name precisely than leave as This PR makes an existing FAQ answer untrue.
After this merges that is the exact case that gains speaker labels - the mono path is the It is also the natural home for the four-slot limit, which currently lives only in the PR's
Two reasons for putting it in this PR rather than an issue: it ships with the feature that Not a blocker from me either way. If you would rather keep this PR code-only, say so and I |
|
Heads-up on CI, since I do not think this one has been looked at: The line that fails, const tailSeconds = Math.max(1.5, Math.min(3.0, micBoundarySeconds * 0.6));
expect(tailSeconds).toBeLessThan(micBoundarySeconds);The floor and the invariant contradict each other: But the boundary being 0.5 s is the finding that matters.
I ran the same What does matter is that So fixing the formula alone would make the spec pass without testing anything - the WAV None of this touches the substance of the PR, which I still think is the right backend |
|
Heads-up before you rebase this onto current Mechanism, read off the screen rather than inferred: after Worth a look beyond the test: a genuinely failed recording whose session identity settles late would show |
Adds a macOS-only Swift/CoreML sidecar (diarize-sidecar, wrapping FluidAudio's Sortformer) that diarizes the mic and system-audio channels independently, so multiple speakers sharing one side of a call (in-person conversations on mic, or multiple remote participants on system audio) get labelled "You" / "Speaker 2" / "Speaker 3" instead of being lumped together. The dominant-by-duration cluster on each channel keeps the legacy "You"/ "Others" label; other clusters are numbered by first chronological appearance across both channels. Any failure (missing binary, timeout, bad output, single-cluster result) falls back to today's exact channel-only behaviour, so this can never fail a meeting. Two fixes from testing against a real in-person recording: - is_diarised now reflects whether the output actually has more than one speaker label, not whether both channels had content — the old check discarded the whole labelled transcript whenever one channel (e.g. system audio with nothing playing) was empty. - Long Parakeet sentences that span multiple real diarizer turns (no strong punctuation break in a long run of speech) are now split at the word level and reassigned per-word, instead of forcing the entire block onto whichever diarizer segment the sentence's midpoint happened to land in.
Mono recordings (many imports — phone voice memos, single-track exports) had no channel split to fall back on, so they got zero speaker labelling, even when the track genuinely has multiple speakers. Runs steno-diarize directly against the whole file instead, reusing the same per-channel tagging/placeholder-resolution helpers the stereo path uses, treating the single track as the "You" channel — consistent with the pre-diarization convention of attributing an unlabelled mono recording to the user. A single real speaker still produces a plain, unlabelled transcript exactly as before.
Sortformer was configured with .default (fastV2_1: 0.48s of audio per CoreML invocation, ~1.04s latency) -- tuned for live/streaming responsiveness this app has no use for, since diarization only ever runs on a fully-recorded, already-finished channel. Switched to .highContextV2 (27.2s per invocation) for recordings long enough to benefit: ~56x fewer invocations for the same audio (~400 vs ~22,500 for a 3-hour file). Measured on a real ~21-minute recording: 153s -> 23s. V2, not V2.1: FluidAudio's own docs note V2.1 "may degrade when many speakers are talking simultaneously" -- a real risk given this app's crosstalk/echo findings from earlier diarization work. Real regression found and fixed during validation: highContextV2's chunk loader requires a full ~30.4s window before it emits anything at all -- a 12.25s test clip came back with zero segments. Added sortformerHighContextMinDuration (90s, real margin above the hard minimum) so recordings shorter than that keep using .default. Accuracy validated against the same real file, not just "it still runs": 98.2% agreement on the dominant speaker's per-second attribution, a near-zero spurious "4th speaker" (0.6s total) cleanly disappeared, and total detected speech stayed within ~1%. The GPU-vs-ANE compute units env var wired into the manual/backfill CLI paths in a prior commit (measured separately: 23.0s ANE vs 18.0s GPU on the same file) stays opt-in-only -- the normal per-meeting pipeline keeps the power/thermal-efficient ANE default.
Ports the diarization-relevant slice of 04c2be1 (which mixed diarization,
progress, and identity concerns in one commit) onto the standalone
diarization branch, minus everything identity-specific:
- CHANNEL_DOMINANCE_THRESHOLD: a channel with one overwhelmingly dominant
speaker is treated as single-speaker rather than spawning a phantom
second speaker from a misdiarization blip.
- CHANNEL_DETECT_TIMEOUT_S: the channel-count probe's fixed 15s timeout
silently dropped long WebM recordings to mono; scaled to 60s.
- _run_steno_diarize rewritten to Popen + two reader threads (avoids the
classic pipe-deadlock on large stdout payloads) with a real JSON
decoder scan for the last valid segment array, tolerating FluidAudio/
CoreML warning text before, between, or after the payload.
- _heartbeat_while_waiting + PROGRESS:diarize:{label}:start/:done so a
long diarization pass doesn't look hung to Electron's inactivity
watchdog or sit on a static spinner.
- Pre-processing audio start log line, so loudnorm's two-pass analysis
doesn't look like a hang on a long recording either.
The sidecar's Output contract stays a bare segment array (this branch
never extracts voiceprint embeddings), so the parser and its tests are
array-only rather than the array-or-object form the full identity branch
needs.
Renders the PROGRESS:diarize:{label}:start/:done markers (added to the
backend in the previous commit) as a real UI stage instead of a static
"Analyzing transcript" spinner sitting through a diarization pass that
can run for minutes on a long recording.
- New 'diarizing' stage with an elapsed-time ticker (this branch's
sidecar has no per-chunk checkpoint to report a percentage from, so
a plain "(Ns)" counter is the only way to show the stage is alive).
- Fixes a real bug the new diarize progress lines would otherwise hit:
the processingProgress handler used to key off ANY PROGRESS: line
unconditionally to flip transcribing -> summarizing; without a
prefix check, a PROGRESS:diarize:* line would have prematurely
jumped the stage to "summarizing" while diarization was still
running. Now branches on the PROGRESS:summarize:/PROGRESS:diarize:
prefix explicitly.
- Clears chunkProgress on every stage transition (summarize-complete,
processing-complete, retry) so a stale diarizing/summarizing
sub-label can't leak into finalizing/error/a retried run.
- main.js: PROGRESS:diarize:* markers now persisted to the on-disk
pipeline log (already true for HEARTBEAT); the live renderer forward
needed no change since the existing PROGRESS: forwarder is generic.
- New processing-stages.t1.spec.ts (mock IPC, real webContents.send
events) -- Processing.tsx had zero test coverage before this.
canRetry (Processing.tsx) requires both retryAudioFile (from processing-complete's audioFile field) and activeSession (from recording.sessionName) to be truthy. The spec reached /meetings/processing via a bare URL hash with no active mock recording, so activeSession stayed null and the retry-button assertion hung waiting on a permanently-disabled button. Start a mock recording first, matching how the screen is actually reached in real usage, and include audioFile in the failure payload.
…invariant CI found this failing 3/3 times on the macOS T2 pipeline lane, not a flake. isSayAvailable() only probed `say -v ?` (listing voices), which exits 0 even on a runner where `say` can't actually synthesize speech -- observed producing ~170 bytes of near-silent PCM (~5ms) instead of real audio, presumably missing voice assets. That let micBoundarySeconds come out at ~0.5s instead of the several seconds a real sentence takes, which fed straight into a tailSeconds formula with an unconditional 1.5s floor (Math.max(1.5, ...)) that made the very next assertion (tailSeconds < micBoundarySeconds) mathematically impossible to satisfy below a 1.5s boundary. Passed locally only because real speech synthesis on a real Mac comfortably exceeds that. Fixes both: isSayAvailable() now synthesizes a short real phrase and measures what it actually wrote, skipping loudly (existing test.skip path) when it's implausibly short, rather than trusting a voice-list probe that doesn't exercise synthesis at all. tailSeconds is now a bounded fraction of micBoundarySeconds with no floor above it, so the invariant holds for any micBoundarySeconds > 0 -- not just relying on the environment guard to keep it out of the impossible range.
This PR makes two existing statements false. docs/faq.mdx's "Can Steno record in-person meetings?" said in-person recordings have no speaker labels at all -- after this PR, the mono/mic-only path is exactly where acoustic diarization gets used, so that's now the headline case that gains labels, not the one that lacks them. docs/features/recording.mdx's "Speaker labels" section had the same gap from the other direction: it described labeling as something that only happens when system audio is on (the [You]/[Others] channel split), omitting the new within-channel acoustic split entirely. Both now describe the real constraint precisely: up to four distinct voices per channel, not four total, since diarization runs independently on each channel. Framed that way because it matters for the common case -- a two-person call is one person per channel, comfortably under the per-channel limit either side, so most users never approach it, but a flat "four speakers" would incorrectly suggest otherwise.
27e2033 to
e129c0a
Compare
|
Retargeted the base to One red check left: T1, The error panel never renders: after Mechanism, and it isn't your code. Evidence rather than reading: patching only I diffed the branch against its merge base I'd fix it on the spec side rather than in production. The reset exists so a screen that stays mounted across back-to-back recordings repaints fresh ( Everything else is green: 11 checks, including both Windows jobs and both pipeline runs. |
|
Here's the spec-side fix, tested rather than suggested this time — on --- a/e2e/specs/processing-stages.t1.spec.ts
+++ b/e2e/specs/processing-stages.t1.spec.ts
@@ -28,6 +28,11 @@ async function openProcessing(page: Page) {
window.location.hash = '/meetings/processing';
});
await expect(page.getByTestId('processing-stage-label')).toBeVisible();
+ // The queue poll that first reports this recording bumps Processing's
+ // `generation`, whose render-phase reset clears stage + retryAudioFile.
+ // Wait until that has landed (the header switches from 'Note' to the
+ // session name) so a later emit can't race the reset.
+ await expect(page.getByRole('heading', { name: 'test-session' })).toBeVisible();
}Why it settles the race: the header is Yours to take or leave, it's your PR. One weaker observation while I was in there: |
…essing-stages.t1
openProcessing() returned before the queue poll's first report of the
mock recording finished bumping Processing.tsx's `generation`, whose
render-phase reset (:143-152) then cleared a caller's terminal stage
(setStage('error')) one round-trip later. Wait for the session name to
paint in the header first, so the reset has already landed before any
event is emitted.
Root-caused and verified (9/9 green, 3 repeats) by Ben/Optic00 on PR stenolabs#455.
… land The generation-settle wait fixed the error-panel race but exposed a different, pre-existing one: the renderer attaches its IPC listeners in a useEffect that runs after the initial paint, so an event sent right after the DOM updates can land in the gap before that effect mounts and be silently dropped (webContents.send has no queueing/replay). CI's runner is apparently slower/more loaded than local, tipping this from theoretical into a real intermittent failure across all three tests in the file. emitUntil() resends on a short interval until the expected UI change is actually observed, rather than firing once and hoping. Safe to repeat: every handler this spec drives is idempotent once its target state is reached, and the loop stops as soon as the check passes.
…til they land" This reverts commit cac5eaa.
|
Correction on the patch I posted above: I verified it on macOS only, and it made things worse. Before it, T1 failed on one test; with it ( The lead I'd chase instead: I am not asserting that yet. I have it running on our own fork right now with |
|
Result, as promised. It is I took your current head ( - const { app, page } = await launchApp({ mockIpc: true });
+ const { app, page } = await launchApp({ mockIpc: true, fakeAudio: true });On the same Ubuntu runner the whole T1 suite goes green — 68 passed, including all three tests in this file: Why: Your revert of Evidence, in case you want to look: run on our fork, branch built from your head with that one change. |
The CI runner has no audio device, so openProcessing()'s recording start fails there and the emitted progress events never take effect. Every other spec in the suite that records already sets this flag. Verified green on the Ubuntu runner from this exact head: 68 passed.
|
As agreed, I've pushed the fix onto this branch rather than leaving it waiting: I've also taken it out of draft. The draft status was my own suggestion from back when this targeted Once CI is green I'll merge it into That should also clean up #472 for you: with the diarization commits in the base branch, merging |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
11 issues found across 20 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="docs/features/recording.mdx">
<violation number="1" location="docs/features/recording.mdx:31">
P3: Docs currently imply acoustic speaker separation applies only when system audio is enabled, which can mislead microphone-only users about available labeling behavior. Consider describing diarization as available on microphone-only recordings too, with channel split only when system audio is present.</violation>
</file>
<file name="docs/faq.mdx">
<violation number="1" location="docs/faq.mdx:93">
P2: This FAQ now promises per-speaker labeling for in-person meetings as a guaranteed behavior, which can mislead Windows users or macOS installs running fallback mode. Consider scoping this to macOS/availability and phrasing as best-effort rather than guaranteed.</violation>
</file>
<file name="stenoai.spec">
<violation number="1" location="stenoai.spec:260">
P2: Non-mac builds can still bundle `steno-diarize` if the file exists, because the new darwin-only `elif` misses and the generic `else` path still includes it under `ollama/`. Consider explicitly handling `base == 'steno-diarize'` and `continue` so non-darwin really skips this sidecar.</violation>
</file>
<file name="e2e/fixtures/say-stereo-wav.ts">
<violation number="1" location="e2e/fixtures/say-stereo-wav.ts:113">
P3: The temp dirs (and the `say`-generated WAV files inside them) created by makeStereoSpeechWav and the isSayAvailable probe are never removed, so every e2e run leaks a fresh `stenoai-e2e-say-*` plus a probe dir under os.tmpdir. That diverges from the repo convention (other specs/fixtures rmSync their temp dirs) and accumulates garbage on long-lived dev machines and CI agents. Wrap the body in a try/finally that calls `rmSync(dir, {recursive:true, force:true})`, and delete the probe dir on the non-throwing path of isSayAvailable.</violation>
</file>
<file name="diarize-sidecar/Sources/main.swift">
<violation number="1" location="diarize-sidecar/Sources/main.swift:218">
P3: Segment ordering can change between runs when two segments have the same start time, because the comparator does not break ties after dictionary-derived iteration order. Adding deterministic tie-breakers (for example `end` then `speakerId`) keeps diarization JSON stable for downstream processing and tests.</violation>
</file>
<file name="src/transcriber.py">
<violation number="1" location="src/transcriber.py:605">
P3: The `if idx is None: continue` guard in the word-splitting branch is unreachable: `_find_nearest_diar_segment` returns `None` only when `diar_segments` is empty, and the function returns early in that case before reaching the token loop. Harmless, but it's dead defensive code that reads as if the loop can handle a missing segment — it can't — which is mildly misleading about the invariant. Consider dropping the guard (or documenting why it's unreachable).</violation>
<violation number="2" location="src/transcriber.py:766">
P2: Malformed diarizer JSON can still crash transcription instead of falling back to legacy labels. The conversion of `raw_segments` to floats is outside failure handling, so bad `start`/`end` fields raise and bypass the safe `None` fallback path.</violation>
<violation number="3" location="src/transcriber.py:1831">
P3: The mono path duplicates the turn-collapse + labelled-transcript assembly that the stereo path already implements. Since both now feed from the same `_tag_channel_segments`/`_resolve_speaker_placeholders` pipeline, extracting a small shared helper (e.g. `turns = _collapse_turns(tagged)` returning `(start, speaker, parts)`) would let both paths reuse the exact same turn-boundary logic and avoid future drift between the two labelling formats. Low severity, but the duplication is real and easy to unify.</violation>
</file>
<file name="e2e/specs/speaker-diarization.t2.spec.ts">
<violation number="1" location="e2e/specs/speaker-diarization.t2.spec.ts:73">
P3: The tail cap at 3.0s only guarantees SPEAKER_0 dominance, but the backend additionally rejects a two-cluster channel via CHANNEL_DOMINANCE_THRESHOLD=0.92 in src/transcriber._cluster_channel_labels. For this fixture, boundary/(boundary+3.0) crosses that gate once micBoundarySeconds exceeds ~34.5s (a sufficiently slow/long run of utterance A), which silently falls back to legacy single-label and fails the [Speaker 2]/[Speaker 3] assertions even though the fixture looks correct. The comment's 'dominant so two clusters pass' reasoning skips that separate, stricter condition. The hardcoded sentence stays well under this today, so it's a latent fragility; consider guarding the invariant explicitly so a future voice/longer sentence doesn't turn into a confusing flake.</violation>
</file>
<file name="app/renderer/src/routes/Processing.tsx">
<violation number="1" location="app/renderer/src/routes/Processing.tsx:580">
P2: The diarize elapsed ticker is not torn down when `generation` changes. If the user starts a new recording while the previous meeting is still in its long diarization stretch, the render-phase reset (setStage('transcribing') + setChunkProgress(null)) runs, but the old `setInterval` from the prior diarization keeps firing and re-writes a stale "Diarizing … channel… (Ns)" label onto the new, freshly-transcribing stage — since the StageCard now shows `chunkProgress` in every non-finalizing/error stage. That's the same category of stale-label leak the feature's tests specifically guard against, just on the generation path instead of the error/finalizing path. Consider clearing the timer in the generation-reset block (alongside `setChunkProgress(null)`), or adding `generation` to the IPC effect's dependency array so the interval is cleaned up on a new generation. Note the reset block runs during render, so a `clearInterval` ref-mutation there is consistent with the existing render-phase state resets.</violation>
</file>
<file name="app/renderer/src/lib/transcriptSegments.ts">
<violation number="1" location="app/renderer/src/lib/transcriptSegments.ts:23">
P2: The diarised-segment regex was widened from an explicit `(You|Others)` to match any `[^\]]+` content, so any bracketed text that appears inside a segment body — not just a real `[Speaker N]` marker — is now treated as a new speaker boundary. For example `[You] Call me at [5:00] tomorrow` is parsed into a phantom `5:00` speaker segment (rendered as an 'Others' bubble), and text after a lone trailing bracket gets dropped. This regresses the previous behavior where bracketed content stayed with its segment. Recommend constraining the marker to the actual label set the pipeline emits (e.g. `(?:You|Others|Speaker \d+)`) in both the captured group and the lookahead, so only genuine markers split segments.</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
|
|
||
| <Accordion title="Can Steno record in-person meetings?"> | ||
| Yes. Steno records from your Mac's microphone, which will pick up voices in the room. For best results, place your laptop centrally. In-person recordings are mic-only, so they have no `[You]` / `[Others]` speaker labels. | ||
| Yes. Steno records from your Mac's microphone, which will pick up voices in the room. For best results, place your laptop centrally. Steno separates up to four distinct voices per recording channel, so an in-person meeting of four or fewer people gets per-speaker labels; beyond that, additional speakers are merged into the four it detects. |
There was a problem hiding this comment.
P2: This FAQ now promises per-speaker labeling for in-person meetings as a guaranteed behavior, which can mislead Windows users or macOS installs running fallback mode. Consider scoping this to macOS/availability and phrasing as best-effort rather than guaranteed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/faq.mdx, line 93:
<comment>This FAQ now promises per-speaker labeling for in-person meetings as a guaranteed behavior, which can mislead Windows users or macOS installs running fallback mode. Consider scoping this to macOS/availability and phrasing as best-effort rather than guaranteed.</comment>
<file context>
@@ -90,7 +90,7 @@ Steno can capture system audio -- the audio playing through your Mac's speakers
<Accordion title="Can Steno record in-person meetings?">
-Yes. Steno records from your Mac's microphone, which will pick up voices in the room. For best results, place your laptop centrally. In-person recordings are mic-only, so they have no `[You]` / `[Others]` speaker labels.
+Yes. Steno records from your Mac's microphone, which will pick up voices in the room. For best results, place your laptop centrally. Steno separates up to four distinct voices per recording channel, so an in-person meeting of four or fewer people gets per-speaker labels; beyond that, additional speakers are merged into the four it detects.
</Accordion>
</file context>
| Yes. Steno records from your Mac's microphone, which will pick up voices in the room. For best results, place your laptop centrally. Steno separates up to four distinct voices per recording channel, so an in-person meeting of four or fewer people gets per-speaker labels; beyond that, additional speakers are merged into the four it detects. | |
| Yes. Steno records from your Mac's microphone, which will pick up voices in the room. For best results, place your laptop centrally. On macOS, Steno can label up to four speakers per recording channel when diarization is available; if unavailable or if more than four people are speaking, labels may fall back or speakers may be merged. |
| if base in ('ffmpeg', 'ffmpeg.exe'): | ||
| # Put ffmpeg at the root of the bundle for easy PATH access | ||
| binaries.append((filepath, '.')) | ||
| elif base == 'steno-diarize' and _IS_DARWIN: |
There was a problem hiding this comment.
P2: Non-mac builds can still bundle steno-diarize if the file exists, because the new darwin-only elif misses and the generic else path still includes it under ollama/. Consider explicitly handling base == 'steno-diarize' and continue so non-darwin really skips this sidecar.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At stenoai.spec, line 260:
<comment>Non-mac builds can still bundle `steno-diarize` if the file exists, because the new darwin-only `elif` misses and the generic `else` path still includes it under `ollama/`. Consider explicitly handling `base == 'steno-diarize'` and `continue` so non-darwin really skips this sidecar.</comment>
<file context>
@@ -257,6 +257,15 @@ if os.path.exists(ollama_bin_dir):
if base in ('ffmpeg', 'ffmpeg.exe'):
# Put ffmpeg at the root of the bundle for easy PATH access
binaries.append((filepath, '.'))
+ elif base == 'steno-diarize' and _IS_DARWIN:
+ # macOS-only Swift/CoreML diarization sidecar (built by
+ # scripts/build-diarize-sidecar.sh). Root-level like ffmpeg
</file context>
| "start": float(s["start"]), | ||
| "end": float(s["end"]), | ||
| "speaker": str(s["speakerId"]), | ||
| } | ||
| for s in raw_segments | ||
| ), | ||
| key=lambda s: s["start"], | ||
| ) | ||
| return _merge_close_diar_segments(segments, STENO_DIARIZE_MERGE_GAP_S) | ||
|
|
||
|
|
||
| def _cluster_channel_labels(diar_segments: list[dict], legacy_label: str) -> Optional[dict[str, str]]: |
There was a problem hiding this comment.
P2: Malformed diarizer JSON can still crash transcription instead of falling back to legacy labels. The conversion of raw_segments to floats is outside failure handling, so bad start/end fields raise and bypass the safe None fallback path.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/transcriber.py, line 766:
<comment>Malformed diarizer JSON can still crash transcription instead of falling back to legacy labels. The conversion of `raw_segments` to floats is outside failure handling, so bad `start`/`end` fields raise and bypass the safe `None` fallback path.</comment>
<file context>
@@ -411,6 +488,400 @@ def _drop_per_segment_bleed(
+ segments = sorted(
+ (
+ {
+ "start": float(s["start"]),
+ "end": float(s["end"]),
+ "speaker": str(s["speakerId"]),
</file context>
| "start": float(s["start"]), | |
| "end": float(s["end"]), | |
| "speaker": str(s["speakerId"]), | |
| } | |
| for s in raw_segments | |
| ), | |
| key=lambda s: s["start"], | |
| ) | |
| return _merge_close_diar_segments(segments, STENO_DIARIZE_MERGE_GAP_S) | |
| def _cluster_channel_labels(diar_segments: list[dict], legacy_label: str) -> Optional[dict[str, str]]: | |
| try: | |
| segments = sorted( | |
| ( | |
| { | |
| "start": float(s["start"]), | |
| "end": float(s["end"]), | |
| "speaker": str(s["speakerId"]), | |
| } | |
| for s in raw_segments | |
| ), | |
| key=lambda s: s["start"], | |
| ) | |
| except (KeyError, TypeError, ValueError) as e: | |
| logger.warning("steno-diarize emitted malformed segment payload: %s", e) | |
| return None | |
| return _merge_close_diar_segments(segments, STENO_DIARIZE_MERGE_GAP_S) |
| style={{ color: 'var(--fg-1)', fontFamily: 'var(--font-sans)' }} | ||
| > | ||
| {chunkProgress && stage === 'summarizing' ? chunkProgress : STAGE_LABEL[stage]} | ||
| {chunkProgress && stage !== 'finalizing' && stage !== 'error' ? chunkProgress : STAGE_LABEL[stage]} |
There was a problem hiding this comment.
P2: The diarize elapsed ticker is not torn down when generation changes. If the user starts a new recording while the previous meeting is still in its long diarization stretch, the render-phase reset (setStage('transcribing') + setChunkProgress(null)) runs, but the old setInterval from the prior diarization keeps firing and re-writes a stale "Diarizing … channel… (Ns)" label onto the new, freshly-transcribing stage — since the StageCard now shows chunkProgress in every non-finalizing/error stage. That's the same category of stale-label leak the feature's tests specifically guard against, just on the generation path instead of the error/finalizing path. Consider clearing the timer in the generation-reset block (alongside setChunkProgress(null)), or adding generation to the IPC effect's dependency array so the interval is cleaned up on a new generation. Note the reset block runs during render, so a clearInterval ref-mutation there is consistent with the existing render-phase state resets.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At app/renderer/src/routes/Processing.tsx, line 580:
<comment>The diarize elapsed ticker is not torn down when `generation` changes. If the user starts a new recording while the previous meeting is still in its long diarization stretch, the render-phase reset (setStage('transcribing') + setChunkProgress(null)) runs, but the old `setInterval` from the prior diarization keeps firing and re-writes a stale "Diarizing … channel… (Ns)" label onto the new, freshly-transcribing stage — since the StageCard now shows `chunkProgress` in every non-finalizing/error stage. That's the same category of stale-label leak the feature's tests specifically guard against, just on the generation path instead of the error/finalizing path. Consider clearing the timer in the generation-reset block (alongside `setChunkProgress(null)`), or adding `generation` to the IPC effect's dependency array so the interval is cleaned up on a new generation. Note the reset block runs during render, so a `clearInterval` ref-mutation there is consistent with the existing render-phase state resets.</comment>
<file context>
@@ -523,10 +572,12 @@ function StageCard({
style={{ color: 'var(--fg-1)', fontFamily: 'var(--font-sans)' }}
>
- {chunkProgress && stage === 'summarizing' ? chunkProgress : STAGE_LABEL[stage]}
+ {chunkProgress && stage !== 'finalizing' && stage !== 'error' ? chunkProgress : STAGE_LABEL[stage]}
</span>
</div>
</file context>
| // the previous one. | ||
| const DIARISED_SEGMENT_RE = | ||
| /(?:\[(\d{1,3}:\d{2}(?::\d{2})?)\]\s*)?\[(You|Others)\]\s*([\s\S]*?)(?=(?:\[\d{1,3}:\d{2}(?::\d{2})?\]\s*)?\[(?:You|Others)\]|$)/g; | ||
| /(?:\[(\d{1,3}:\d{2}(?::\d{2})?)\]\s*)?\[([^\]]+)\]\s*([\s\S]*?)(?=(?:\[\d{1,3}:\d{2}(?::\d{2})?\]\s*)?\[[^\]]+\]|$)/g; |
There was a problem hiding this comment.
P2: The diarised-segment regex was widened from an explicit (You|Others) to match any [^\]]+ content, so any bracketed text that appears inside a segment body — not just a real [Speaker N] marker — is now treated as a new speaker boundary. For example [You] Call me at [5:00] tomorrow is parsed into a phantom 5:00 speaker segment (rendered as an 'Others' bubble), and text after a lone trailing bracket gets dropped. This regresses the previous behavior where bracketed content stayed with its segment. Recommend constraining the marker to the actual label set the pipeline emits (e.g. (?:You|Others|Speaker \d+)) in both the captured group and the lookahead, so only genuine markers split segments.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At app/renderer/src/lib/transcriptSegments.ts, line 23:
<comment>The diarised-segment regex was widened from an explicit `(You|Others)` to match any `[^\]]+` content, so any bracketed text that appears inside a segment body — not just a real `[Speaker N]` marker — is now treated as a new speaker boundary. For example `[You] Call me at [5:00] tomorrow` is parsed into a phantom `5:00` speaker segment (rendered as an 'Others' bubble), and text after a lone trailing bracket gets dropped. This regresses the previous behavior where bracketed content stayed with its segment. Recommend constraining the marker to the actual label set the pipeline emits (e.g. `(?:You|Others|Speaker \d+)`) in both the captured group and the lookahead, so only genuine markers split segments.</comment>
<file context>
@@ -3,28 +3,32 @@
+// the previous one.
const DIARISED_SEGMENT_RE =
- /(?:\[(\d{1,3}:\d{2}(?::\d{2})?)\]\s*)?\[(You|Others)\]\s*([\s\S]*?)(?=(?:\[\d{1,3}:\d{2}(?::\d{2})?\]\s*)?\[(?:You|Others)\]|$)/g;
+ /(?:\[(\d{1,3}:\d{2}(?::\d{2})?)\]\s*)?\[([^\]]+)\]\s*([\s\S]*?)(?=(?:\[\d{1,3}:\d{2}(?::\d{2})?\]\s*)?\[[^\]]+\]|$)/g;
export function parseTranscript(text: string, isDiarised: boolean): Segment[] {
</file context>
| /(?:\[(\d{1,3}:\d{2}(?::\d{2})?)\]\s*)?\[([^\]]+)\]\s*([\s\S]*?)(?=(?:\[\d{1,3}:\d{2}(?::\d{2})?\]\s*)?\[[^\]]+\]|$)/g; | |
| /(?:\[(\d{1,3}:\d{2}(?::\d{2})?)\]\s*)?\[(You|Others|Speaker \d+)\]\s*([\s\S]*?)(?=(?:\[\d{1,3}:\d{2}(?::\d{2})?\]\s*)?\[(?:You|Others|Speaker \d+)\]|$)/g; |
| destPath: string, | ||
| opts: { micUtteranceA: string; micUtteranceB: string; systemUtterance: string }, | ||
| ): StereoSpeechResult { | ||
| const dir = mkdtempSync(path.join(tmpdir(), 'stenoai-e2e-say-')); |
There was a problem hiding this comment.
P3: The temp dirs (and the say-generated WAV files inside them) created by makeStereoSpeechWav and the isSayAvailable probe are never removed, so every e2e run leaks a fresh stenoai-e2e-say-* plus a probe dir under os.tmpdir. That diverges from the repo convention (other specs/fixtures rmSync their temp dirs) and accumulates garbage on long-lived dev machines and CI agents. Wrap the body in a try/finally that calls rmSync(dir, {recursive:true, force:true}), and delete the probe dir on the non-throwing path of isSayAvailable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At e2e/fixtures/say-stereo-wav.ts, line 113:
<comment>The temp dirs (and the `say`-generated WAV files inside them) created by makeStereoSpeechWav and the isSayAvailable probe are never removed, so every e2e run leaks a fresh `stenoai-e2e-say-*` plus a probe dir under os.tmpdir. That diverges from the repo convention (other specs/fixtures rmSync their temp dirs) and accumulates garbage on long-lived dev machines and CI agents. Wrap the body in a try/finally that calls `rmSync(dir, {recursive:true, force:true})`, and delete the probe dir on the non-throwing path of isSayAvailable.</comment>
<file context>
@@ -0,0 +1,127 @@
+ destPath: string,
+ opts: { micUtteranceA: string; micUtteranceB: string; systemUtterance: string },
+): StereoSpeechResult {
+ const dir = mkdtempSync(path.join(tmpdir(), 'stenoai-e2e-say-'));
+
+ const uttA = synthesize(opts.micUtteranceA, path.join(dir, 'mic_a.wav'));
</file context>
| end: Double(seg.endTime) | ||
| ) | ||
| } | ||
| .sorted { $0.start < $1.start } |
There was a problem hiding this comment.
P3: Segment ordering can change between runs when two segments have the same start time, because the comparator does not break ties after dictionary-derived iteration order. Adding deterministic tie-breakers (for example end then speakerId) keeps diarization JSON stable for downstream processing and tests.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At diarize-sidecar/Sources/main.swift, line 218:
<comment>Segment ordering can change between runs when two segments have the same start time, because the comparator does not break ties after dictionary-derived iteration order. Adding deterministic tie-breakers (for example `end` then `speakerId`) keeps diarization JSON stable for downstream processing and tests.</comment>
<file context>
@@ -0,0 +1,232 @@
+ end: Double(seg.endTime)
+ )
+ }
+ .sorted { $0.start < $1.start }
+
+ let encoded = try JSONEncoder().encode(segments)
</file context>
| .sorted { $0.start < $1.start } | |
| .sorted { ($0.start, $0.end, $0.speakerId) < ($1.start, $1.end, $1.speakerId) } |
| const tailSeconds = Math.min(micBoundarySeconds * 0.6, 3.0); | ||
| expect(tailSeconds).toBeLessThan(micBoundarySeconds); | ||
|
|
There was a problem hiding this comment.
P3: The tail cap at 3.0s only guarantees SPEAKER_0 dominance, but the backend additionally rejects a two-cluster channel via CHANNEL_DOMINANCE_THRESHOLD=0.92 in src/transcriber._cluster_channel_labels. For this fixture, boundary/(boundary+3.0) crosses that gate once micBoundarySeconds exceeds ~34.5s (a sufficiently slow/long run of utterance A), which silently falls back to legacy single-label and fails the [Speaker 2]/[Speaker 3] assertions even though the fixture looks correct. The comment's 'dominant so two clusters pass' reasoning skips that separate, stricter condition. The hardcoded sentence stays well under this today, so it's a latent fragility; consider guarding the invariant explicitly so a future voice/longer sentence doesn't turn into a confusing flake.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At e2e/specs/speaker-diarization.t2.spec.ts, line 73:
<comment>The tail cap at 3.0s only guarantees SPEAKER_0 dominance, but the backend additionally rejects a two-cluster channel via CHANNEL_DOMINANCE_THRESHOLD=0.92 in src/transcriber._cluster_channel_labels. For this fixture, boundary/(boundary+3.0) crosses that gate once micBoundarySeconds exceeds ~34.5s (a sufficiently slow/long run of utterance A), which silently falls back to legacy single-label and fails the [Speaker 2]/[Speaker 3] assertions even though the fixture looks correct. The comment's 'dominant so two clusters pass' reasoning skips that separate, stricter condition. The hardcoded sentence stays well under this today, so it's a latent fragility; consider guarding the invariant explicitly so a future voice/longer sentence doesn't turn into a confusing flake.</comment>
<file context>
@@ -0,0 +1,129 @@
+ // where `say` produced near-silent audio (micBoundarySeconds well under
+ // 1.5s). isSayAvailable() now measures real synthesized duration, so
+ // that shouldn't recur, but this formula no longer depends on it either.
+ const tailSeconds = Math.min(micBoundarySeconds * 0.6, 3.0);
+ expect(tailSeconds).toBeLessThan(micBoundarySeconds);
+
</file context>
| const tailSeconds = Math.min(micBoundarySeconds * 0.6, 3.0); | |
| expect(tailSeconds).toBeLessThan(micBoundarySeconds); | |
| const tailSeconds = Math.min(micBoundarySeconds * 0.6, 3.0); | |
| // Keep the two-cluster ratio below CHANNEL_DOMINANCE_THRESHOLD (0.92) so the | |
| // backend's _cluster_channel_labels doesn't silently downgrade to legacy. | |
| expect(tailSeconds).toBeGreaterThan(micBoundarySeconds * 0.09); | |
| expect(tailSeconds).toBeLessThan(micBoundarySeconds); |
| t_start = float(token.get("start") or 0.0) | ||
| t_end = float(token.get("end") or t_start) | ||
| idx = _find_nearest_diar_segment(t_start, t_end, diar_segments) | ||
| if idx is None: |
There was a problem hiding this comment.
P3: The if idx is None: continue guard in the word-splitting branch is unreachable: _find_nearest_diar_segment returns None only when diar_segments is empty, and the function returns early in that case before reaching the token loop. Harmless, but it's dead defensive code that reads as if the loop can handle a missing segment — it can't — which is mildly misleading about the invariant. Consider dropping the guard (or documenting why it's unreachable).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/transcriber.py, line 605:
<comment>The `if idx is None: continue` guard in the word-splitting branch is unreachable: `_find_nearest_diar_segment` returns `None` only when `diar_segments` is empty, and the function returns early in that case before reaching the token loop. Harmless, but it's dead defensive code that reads as if the loop can handle a missing segment — it can't — which is mildly misleading about the invariant. Consider dropping the guard (or documenting why it's unreachable).</comment>
<file context>
@@ -411,6 +488,400 @@ def _drop_per_segment_bleed(
+ t_start = float(token.get("start") or 0.0)
+ t_end = float(token.get("end") or t_start)
+ idx = _find_nearest_diar_segment(t_start, t_end, diar_segments)
+ if idx is None:
+ continue
+ if run_index is not None and idx != run_index:
</file context>
| tagged = _resolve_speaker_placeholders(tagged) | ||
|
|
||
| turns: list[tuple[float, str, list[str]]] = [] | ||
| for start, speaker, text in tagged: |
There was a problem hiding this comment.
P3: The mono path duplicates the turn-collapse + labelled-transcript assembly that the stereo path already implements. Since both now feed from the same _tag_channel_segments/_resolve_speaker_placeholders pipeline, extracting a small shared helper (e.g. turns = _collapse_turns(tagged) returning (start, speaker, parts)) would let both paths reuse the exact same turn-boundary logic and avoid future drift between the two labelling formats. Low severity, but the duplication is real and easy to unify.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/transcriber.py, line 1831:
<comment>The mono path duplicates the turn-collapse + labelled-transcript assembly that the stereo path already implements. Since both now feed from the same `_tag_channel_segments`/`_resolve_speaker_placeholders` pipeline, extracting a small shared helper (e.g. `turns = _collapse_turns(tagged)` returning `(start, speaker, parts)`) would let both paths reuse the exact same turn-boundary logic and avoid future drift between the two labelling formats. Low severity, but the duplication is real and easy to unify.</comment>
<file context>
@@ -1305,6 +1799,52 @@ def transcribe_diarised(self, audio_filepath: Path, language: str = "en") -> Opt
+ tagged = _resolve_speaker_placeholders(tagged)
+
+ turns: list[tuple[float, str, list[str]]] = []
+ for start, speaker, text in tagged:
+ if turns and turns[-1][1] == speaker:
+ turns[-1][2].append(text)
</file context>
The diarization work landed on the shared branch as the rebased commits (stenolabs#455), while this branch was cut from their pre-rebase originals, so the same changes arrived twice with different ancestry. Every conflicting hunk was that duplication and is resolved to this branch's side; the merge then introduced a second copy of formatElapsedSeconds, which is removed again. Genuinely new from the shared branch, and the only net change here: the processing-stages spec now launches with fakeAudio (the CI runner has no audio device) and waits for the generation bump to settle before emitting.
2 participants
Description
Adds acoustic speaker diarization for both stereo (mic/system) and mono recordings, splitting multiple speakers sharing one side of a call or a single mic into distinct labeled turns instead of the current flat "You"/"Others" split. This picks up issue #359, scoped down to diarization only. A separate follow-up PR covers human-confirmed cross-recording speaker identification (naming a "Speaker 2" from a past meeting), which depends on this one and is a substantially larger review surface (persistent identity storage, matching quality caveats) that deserves its own slower pass.
Diarization runs through a new Swift/CoreML sidecar (
diarize-sidecar/) wrapping FluidAudio's Sortformer diarizer, invoked from Python (src.transcriber._run_steno_diarize), never from Electron, since the batch pipeline is entirely Python-orchestrated. It is macOS-only and gated instenoai.specon both the platform and the sidecar binary existing, so a checkout that skips the Swift build (or Windows/Linux) falls back to the existing channel-only "You"/"Others" labeling automatically. Diarization can never fail a meeting: any failure (missing binary, timeout, bad output, a single-cluster result) degrades to the legacy behavior byte-for-byte.Also adds a live per-stage progress indicator ("Identifying speakers" with an elapsed-time counter) so a long diarization pass on a multi-hour recording no longer sits on a static "Analyzing transcript" spinner with no feedback, and a heartbeat mechanism so Electron's inactivity watchdog does not kill a long-running diarization call.
Type of Change
Testing
python -m unittest discover tests), including new coverage for the sidecar's JSON parsing (real-world quirks: FluidAudio/CoreML warning text before, between, or after the payload; a dominant-speaker gate to avoid spawning phantom speakers from misdiarization blips) and the progress/heartbeat plumbing.ruff checkclean on every file this PR touches.tsc --noEmitclean,eslintclean (0 errors) on the new Processing stage.processing-stages.t1.spec.tscovering the stage-transition logic end to end.speaker-diarization.t2.spec.ts: synthesizes real speech via macOSsayso Parakeet/whisper.cpp produce real ASR segments, points the sidecar at a fixture script, and asserts the saved transcript's per-channel labeling and cross-channel speaker numbering.Additional Notes
Additional Notes
Sortformer has a fixed 4-speaker-slot architecture (no speaker-count hint can be passed to it). This is a known limitation flagged in #359 and its discussion; the identity follow-up PR's discussion covers a possible "who is in this meeting" field as a future lever, which is out of scope here.
The rendered transcript UI (chat bubbles) itself is unchanged by this PR, and it already benefits directly: when a channel has two diarized speakers, e.g. two people sharing one mic with no separate system-audio channel, both show up as two visually distinct bubbles (the dominant speaker as "You", green/right; the other as grey/left), where today they would be merged into one "You" bubble. That per-channel split is the actual feature.
Its limit is at three-plus: the bubble styling only has two visual buckets ("You" vs everyone else), so if diarization finds a third or fourth distinct speaker, e.g. three people sharing one mic, or extra speakers split across both channels, the transcript text still correctly carries distinct "Speaker 2"/"Speaker 3"/"Speaker 4" labels, but the bubbles do not visually differentiate them from each other, or from "Others" — all render identically grey/left, unless a speaker has been given a confirmed real name (the identity follow-up PR). That richer per-speaker label data is already useful today regardless (readable in the raw transcript text/export, and it's what the identity PR's suggestion engine keys off of); giving 3+ distinct speakers their own visual treatment in the bubble UI is a reasonable follow-up, not something this PR does.
Summary by cubic
Adds acoustic speaker diarization for stereo and mono recordings on macOS via
diarize-sidecar(Sortformer), labeling turns as “You” and “Speaker N” instead of channel-only. Adds an “Identifying speakers” stage with an elapsed timer; diarization is optional and always falls back to legacy labels if unavailable or low-confidence.New Features
diarize-sidecar(Swift/CoreML) usingFluidAudioSortformer; invoked from Python and bundled when present, otherwise auto-fallback to “You”/“Others”.PROGRESS:diarize:*and a heartbeat; new “Identifying speakers” stage with an elapsed time counter; progress lines are persisted in the processing log.Bug Fixes
PROGRESS:diarize:*for summarization, clear stale sub-labels on transitions, and persist diarize progress markers.Written for commit 05679be. Summary will update on new commits.