Budgeting latency across a local STT → LLM → TTS pipeline

Context

A voice assistant where no audio leaves the machine: WebRTC in, voice activity detection, speech-to-text, a small local language model, text-to-speech, audio back out. Built to run entirely on one box.

Problem

In a conversation the user feels every delay. The instinct is to assume the language model is responsible for most of it and start optimising there.

Constraints

• The conversations are clinical, so the audio cannot be sent to a third-party provider at all. Keeping it on the machine removes the question instead of having to answer it.
• Per-minute cloud pricing does not work when the assistant is open for a whole shift. The number that matters is minutes per day.
• Everything runs on one machine, so the stages compete for the same hardware.

Decision

Measure every stage separately before optimising any of them, and treat the whole pipeline as a time budget with named line items rather than one number.

The measurements reversed the assumption. Transcribing 2.1 seconds of speech took 1.24 seconds; generating the reply took 1.08. The model was the cheap part. Optimising it first would have spent effort on the second-biggest cost.

Alternatives

Running the speech stages in the cloud would probably be faster per call. It was ruled out on the data requirement first and cost second — a meter that runs whether or not anyone is talking, versus a machine you pay for once.

Trade-offs

Running it locally means we own model versions, hardware capacity and the ways both can fail. In exchange the data genuinely never leaves the machine — rather than being covered by a contract — and a recurring per-minute cost becomes a fixed one.

Outcome

A working end-to-end local pipeline with a per-stage budget, and a correctly identified bottleneck.