Voice recognition transcription: what an action-bound transcript needs that a notes-app one doesn't

Two jobs that share a name

Most pages about voice recognition transcription assume the destination is a text file. You speak, words land in a document, a human cleans them up later. The pipeline is a one-way pipe: audio in, prose out, done.

A growing share of the pipelines I see deployed do not work that way at all. The transcript is not the product. The transcript is a prompt that, a few milliseconds later, will be parsed by an LLM and used to drive UI moves on a real desktop: clicks, keystrokes, scrolls, accessibility actions. The user's intent (“reply to that email”, “send the file to Marwan”, “open the last Notion page I had open this morning”) gets satisfied by an agent that reads the macOS accessibility tree and acts inside the app you already have focused.

Both jobs look identical from one foot away, audio streams in, text streams out, but at six inches the transcription layer has to make different choices for each one. The rest of this page is about those choices, drawn from the actual production code of one such agent.

The pipeline, end to end

Here is the actual call graph between the user holding the Option key and the agent performing an action. Every leg of this is in the public repo at github.com/mediar-ai/fazm and the file names are real.

The audio buffer size, 3200 bytes, is exactly 100 ms of 16-bit mono PCM at 16 kHz (16000 samples per second times 2 bytes per sample times 0.1 s). It lives as private let audioBufferSize = 3200 on line 119 of TranscriptionService.swift. The buffer flushes the moment it fills, so latency from microphone to first chunk on the wire is bounded at ~100 ms regardless of how chatty the user is.

Choice #1: vocabulary bias for product nouns

Deepgram Nova-3 lets a client pass keyterm= query parameters to bias the language model toward specific tokens. The hard cap is 500 terms, but a comment in Fazm's source says effectiveness drops past roughly 30, so the shipped seed list is deliberately small and curated. Every name on it is a token the model otherwise mishears with high probability and that has near-zero collision with common English vocabulary.

The list is rendered straight from the Swift code, in DeletedTypeStubs.swift near line 657 as static let systemVocabulary: [String]. Users add their own terms from a Dictionary settings panel; user terms go first, then de-duplicated system terms get appended. Anything the user removes from the system list gets persisted into disabledSystemVocabulary so an upgrade does not silently restore a term they killed on purpose.

Choice #2: spoken-form rewriting

A notes app can leave “dot com” alone, the writer will fix it. An agent that is about to type a URL into a browser cannot. Deepgram exposes a replace=spoken:written parameter that runs find-and-replace inside the model output. Fazm passes the table below on every English (or multi-language) connection.

The rules only get attached when the language is en or the auto-detect mode multi. Pushing “dot com” -> “.com” on a German or Russian transcript would corrupt unrelated words. That conditional is right at the bottom of private func connect() in TranscriptionService.swift, around line 301.

Choice #3: separating the user from the meeting

When the agent is supposed to listen during a meeting (not just to the operator), the WebSocket is opened with channels=2 and multichannel=true. Channel 0 carries the mic, channel 1 carries the loopback of system audio (the other people's voices from Zoom, Meet, FaceTime, whatever). Deepgram returns segments tagged with a channel_index array, and Fazm extracts channelIndex = response.channel_index?.first ?? 0 into the transcript struct.

Push-to-talk uses channels=1 instead. The whole prompt is yours, the agent does not need to know who else was on the call, and a single-channel stream is cheaper. The two configurations share the same TranscriptionService class; the only thing the calling code does differently is pass channels: 1 or channels: 2 at init time.

Choice #4: surviving the silent dropout

The thing nobody warns you about with streaming transcription: the server-side connection can die silently. The socket reports as open, you keep flushing frames, nothing comes back. The textbook answer is to use the TCP keepalive, but TCP keepalive defaults on macOS are measured in hours, which is several lifetimes for a live caption.

Fazm runs two cooperating timers on top of the WebSocket. Every state transition below comes from named constants near the top of TranscriptionService.swift.

Choice #5: interim vs final

A naive client either renders every segment Deepgram sends (caption jitters wildly as the model revises its guess) or waits for the final segment of an utterance (the caption looks frozen for two seconds). Both feel broken.

Fazm splits the difference. Interim segments (is_final=false) paint a temporary caption strip. The strip clears and gets replaced when the same segment arrives again with is_final=true. The prompt only commits to the LLM on speech_final=true (the model thinks the user finished speaking) OR when the user releases the Option key (push-to-talk explicit end). The endpointing query parameter is set to 300 ms and utterance_end_ms to 1000 ms, so the model uses voice activity detection (300 ms of silence -> segment boundary) as the primary signal and a 1-second all-silence fallback as the backup.

What this means if you are building your own

You can take a generic speech-to-text API and ship a working dictation field in an afternoon. The boring parts are what take the next two weeks: the vocabulary list, the spoken-form rewrites, the channel split, the keepalive-plus-watchdog pair, the interim/final commit logic, and the buffering scheme that keeps latency at ~100 ms without drowning the wire in 20 ms packets. None of those are in the API docs. Most are not in any product's marketing page either, because they sit one layer down from the feature description.

If you are evaluating a voice-driven product, those are the seams to look for. Ask the vendor what their seed vocabulary is. Ask whether spoken URLs come back normalized. Ask how they detect a silent dropout. The answer either exists (in their git history if they are open source, in the engineer's head if not) or the product will have a frustrating quality floor that does not improve with a model upgrade.

For Fazm specifically, every constant in this post is grep-able in the public repo. If something here is wrong or out of date, the file path tells you exactly where to send the pull request.