How much latency and memory does an open-model voice pipeline cost on a CPU, and what reply time does it leave?
research/ambient-pipeline
CPU voice pipeline (Silero VAD, Whisper tiny.en, Qwen2.5-0.5B, Piper) on synthetic speech: per-stage latency/RTF/memory and a reply-latency budget of 0.9–1.5 s at 200–800 ms endpoint hangover, ~0.7 s of it compute. Server CPU, not a phone.
What it found
A streaming Silero VAD, Whisper tiny.en (int8), Qwen2.5-0.5B (4-bit) and Piper chain on four CPU cores, on six synthetic sentences. VAD costs 0.15 ms per 32 ms frame, ASR 306 ms per segment (RTF 0.11), the first LLM sentence 267 ms, the first TTS chunk 88 ms, and the models add about 740 MB. End of speech to first audio is 0.94–1.55 s at 200–800 ms endpoint hangover, of which about 0.7 s is compute; a 200 ms hangover already cut one utterance in two, and a 0.5 s target is impossible once the hangover alone exceeds it. Stylised: synthetic clean speech, one server CPU not a phone, batch ASR, LLM answer quality not evaluated.
A figure from its own output
Paper, code and results
More in Ambient AI assistant
- When should an always-on assistant speak, and does a policy tuned in a simulator keep its annoyance budget in a real room?When an always-on assistant should speak: wake, keyword, threshold and learned policies at equal annoyance, and what breaks when the simulator's room differs from the real one
- How much does speech-recognition error hurt an always-on assistant's trigger policies at equal annoyance?How much speech-recognition error hurts an always-on assistant's trigger policies at equal annoyance: from word error rate 0 to 0.5 a two-word wake phrase loses 74% of its recall and a transcript-reading threshold 76%, an acoustic-only score about 54%, and the transcript policy falls behind the acoustic one at WER 0.3 to 0.4 if its separation shrinks as (1-WER)^2 (stylised).
- When moments of need cluster in time, does a cooldown, a boost or a back-off help an always-on assistant at equal annoyance?When moments of need cluster in time, does a cooldown, boost or back-off help an always-on assistant at equal annoyance: an episode-aware boost gains up to 0.05 recall under strong clustering, a cooldown always loses recall, back-off is a wash; stylised simulation.
- Under a storage limit, which transcripts should an on-device assistant keep, and what survives?What an on-device assistant should keep from a week of simulated transcripts: BM25 retrieval, extractive summaries and byte-limited retention policies (stylised)
- Which parts of an always-on Android listening service can be tested without a phone?Testable Kotlin core of an always-on listening service (re-framing, energy VAD, utterance segmenter, headset-drop policy; 7 tests pass under a shim) plus an unbuilt Android foreground-service sketch; no SDK, no device numbers
- When is a glance better than speech for an always-on assistant?When a glanceable display beats speech for an always-on assistant: a stylised utility model over noise, company, free eyes and message length, with sensitivity and twin-gap checks
All 46 Ambient AI assistant projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.