Research topic
Ambient AI assistant
An always-on, audio-first assistant: when it should speak, and how fast and cheaply the pipeline runs on a device. Simulation first, no real recordings.
46 projects, each with its own page, code and tests. See them on the full research index.
research/ambient-pipeline · 7 tests
CPU voice pipeline (Silero VAD, Whisper tiny.en, Qwen2.5-0.5B, Piper) on synthetic speech: per-stage latency/RTF/memory and a reply-latency budget of 0.9–1.5 s at 200–800 ms endpoint hangover, ~0.7 s of it compute. Server CPU, not a phone.
Abstract
A streaming Silero VAD, Whisper tiny.en (int8), Qwen2.5-0.5B (4-bit) and Piper chain on four CPU cores, on six synthetic sentences. VAD costs 0.15 ms per 32 ms frame, ASR 306 ms per segment (RTF 0.11), the first LLM sentence 267 ms, the first TTS chunk 88 ms, and the models add about 740 MB. End of speech to first audio is 0.94–1.55 s at 200–800 ms endpoint hangover, of which about 0.7 s is compute; a 200 ms hangover already cut one utterance in two, and a 0.5 s target is impossible once the hangover alone exceeds it. Stylised: synthetic clean speech, one server CPU not a phone, batch ASR, LLM answer quality not evaluated.
research/ambient-interject · 9 tests
When an always-on assistant should speak: wake, keyword, threshold and learned policies at equal annoyance, and what breaks when the simulator's room differs from the real one
Abstract
Simulated 10 s listening windows (needs at 3%, an intent score with d′ = 2, wake-phrase and keyword signals) compare interruption policies at equal annoyance. A wake word is precise (0.69) but catches 15% of needs; adding a keyword reaches 53% recall at 28.7 false interruptions per hour (0.72 for the wake word). At a budget of 1 false per hour a score threshold reaches recall 0.26 and a logistic policy that also uses the keyword and wake flag 0.33; recall at that budget climbs from 0.17 to 0.82 as d′ goes from 1 to 4. Tuned to 1 false per hour, policies keep the budget when only the need side shifts (weaker detector, rarer needs) but run at 3.6–3.9 per hour when the no-need background shifts (noisier score, more wake false alarms); 16 labelled hours restore it for the noisier score, and no threshold can for a wake detector that alone causes 3.5 false per hour. Stylised: independent windows, Gaussian scores, chosen costs, no real conversations.
research/ambient-asrnoise · 6 tests
How much speech-recognition error hurts an always-on assistant's trigger policies at equal annoyance: from word error rate 0 to 0.5 a two-word wake phrase loses 74% of its recall and a transcript-reading threshold 76%, an acoustic-only score about 54%, and the transcript policy falls behind the acoustic one at WER 0.3 to 0.4 if its separation shrinks as (1-WER)^2 (stylised).
Abstract
The ambient-interject world with an ASR word error rate (WER) entering each signal by assumption: a two-word wake phrase needs both words right, a keyword one, a transcript-reading intent score loses separation as (1−WER)k, an acoustic score is unaffected. At 2 false interruptions per hour, going from WER 0 to 0.5 cuts the wake phrase's recall from 0.151 to 0.039 (−74%) and the transcript threshold's from 0.365 to 0.089 (−76%), while the acoustic threshold falls from 0.189 to 0.087. With k = 1 the transcript policy stays marginally ahead at WER 0.5; with k = 2 the acoustic policy overtakes it at WER 0.25–0.4. Thresholds tuned at WER 0 stay under the false-interruption budget when deployed at higher WER. Stylised: the WER-to-signal mapping is assumed, not fitted to real recognition, and ASR confidence is not modelled.
research/ambient-cluster · 7 tests
When moments of need cluster in time, does a cooldown, boost or back-off help an always-on assistant at equal annoyance: an episode-aware boost gains up to 0.05 recall under strong clustering, a cooldown always loses recall, back-off is a wash; stylised simulation.
Abstract
Needs are made to cluster with a hidden episode/quiet chain (overall rate fixed at 3% of 10 s windows), and a plain score threshold is compared with three stateful rules at the same budget of false interruptions per hour, paired over 10 seeds. With strong clustering an episode-aware boost (lower the threshold after an accepted answer) raises recall by 0.039 at 1 false/h and 0.054 at 4 (0.467 to 0.521); a cooldown after speaking loses recall everywhere, up to 0.066; a back-off after a dismissal is within noise or slightly negative. Without clustering nothing helps. At the 1 false/h budget the boost also overshoots the budget slightly (1.19 vs 1.11). Stylised: false alarms are independent of the episode state, feedback is exact and immediate, every distribution is chosen, no real conversations.
research/ambient-memory · 9 tests
What an on-device assistant should keep from a week of simulated transcripts: BM25 retrieval, extractive summaries and byte-limited retention policies (stylised)
Abstract
Seven simulated days of 1500 utterances each with 6 planted facts per day. BM25 finds every fact (hit@5 = 1.00) when the key words are rare, which is the easy case. With the store limited to 10% of the week, dropping oldest or random utterances keeps 0.11 and 0.10 of the facts; keeping the highest mean-IDF utterances keeps 0.93, because the facts are rare-word lines by construction. An importance-ranked extractive summary at 5% of a day holds 85% of its facts, against 3% for the first lines. Index build 0.04 s, query 1.9 ms on one container CPU. Stylised: synthetic words, lexical queries, no paraphrase, no speech errors. Produced with AI assistance.
research/ambient-android
Testable Kotlin core of an always-on listening service (re-framing, energy VAD, utterance segmenter, headset-drop policy; 7 tests pass under a shim) plus an unbuilt Android foreground-service sketch; no SDK, no device numbers
Abstract
A plain-Kotlin core (frame re-chunking, an energy-gate voice detector with hangover and pre-roll, an utterance segmenter that never emits silence, and a headset route policy that pauses rather than falling back to the phone mic) with 7 unit tests, which pass when compiled with Kotlin 2.0.21 and run through a test shim. The Android foreground-service shell is unbuilt: no SDK was usable, so no latency, battery or accuracy numbers exist. Synthetic square-wave frames only. Produced with AI assistance.
research/ambient-glance · 10 tests
When a glanceable display beats speech for an always-on assistant: a stylised utility model over noise, company, free eyes and message length, with sensitivity and twin-gap checks
Abstract
A stylised expected-utility model chooses between speech, a glanceable display and silence by noise, other talkers, free eyes and message length; a Monte Carlo check agrees with the closed form to 0.001. A glance wins in about half of 84 grid contexts, and the look rate, the number a simulator is least able to supply, drives the twin gap (up to 0.036 utility per message). Every parameter is a round figure of mine: nothing is fitted or measured.
research/ambient-urgency · 9 tests
Urgency classes and a chime-plus-glance mode added to the glance-vs-speech utility model for an always-on assistant: a stylised model with sensitivity and twin-gap checks
Abstract
A stylised expected-utility model adds three urgency classes (deadlines of 120, 20 and 6 s) and a chime-plus-glance mode to the speech, glance and silence choice. Chime-plus-glance is the best mode in 62% of normal and high-urgency contexts and raises mean utility per message from 0.351 to 0.417 (partly by construction, since high-urgency messages carry twice the weight); a Monte Carlo check agrees with the closed form to 0.002. The look and reaction rates drive the twin gap (up to 0.018 utility per message). Every parameter is a round figure of mine: nothing is fitted or measured.
research/ambient-speculate · 9 tests
How much reply latency starting the ASR/LLM/TTS chain during the end-of-speech hangover hides, from the measured ambient-pipeline stage times and an assumed within-turn pause model (hold-until-endpoint vs eager; extra runs per turn)
Abstract
Using the measured ASR, LLM and TTS times from ambient-pipeline (a 682 ms median chain on one server CPU, six synthetic utterances), the reply pipeline starts after S ms of silence instead of after the full end-of-speech hangover H and the reply is held until H confirms the turn is over. Holding saves exactly min(chain, H − S): at H = 1000 ms the median reply falls from 1744 to 1061 ms, at the price of 0.7 to 1.9 extra pipeline runs per turn depending on S, and the just-in-time start S = H − chain is the cheapest that hides everything. Pause durations inside a turn are assumed (five lognormal settings, not data); with median 400 ms pauses no hangover up to 1.2 s keeps cut-offs under 10%. Closed forms agree with a Monte Carlo check.
research/ambient-semend · 8 tests
Can a sentence-completeness cue let an always-on assistant stop waiting so long before it replies: a stylised two-hangover endpointer cuts the mean wait from 913 to 574 ms at a 10% cut-off budget if no pause sits at a complete point, but only to 837 ms if half do; cue and pauses assumed.
Abstract
A stylised signal-detection model: the endpointer waits a short hangover Hs after a sentence the cue calls complete and a longer Hl otherwise. With medium pauses (2 per turn, median 250 ms, assumed) and a 10% cut-off budget, a fixed hangover is 913 ms; a cue of quality d = 2 cuts the mean wait to 574 ms if no pause sits at a complete point, but only to 837 ms if half of them do, because the cue is blind to those. With long or widely spread pauses, where a fixed hangover is 1.9 to 2.6 s at a 5% budget, d = 3 gives 571 to 873 ms. The cue, the pause model and the share of clause-boundary pauses are assumed, not measured; only the 682 ms reply chain is measured. Closed forms agree with a Monte Carlo check.
research/ambient-rechime · 9 tests
How many repeated earbud tones before a glance are worth their annoyance, by urgency, tone gap and masking, in the ambient-urgency utility model (stylised)
Abstract
A stylised extension of the urgency model lets the assistant repeat a tone up to four times, 3 s apart, each noticed independently and cancelled once the wearer looks. Choosing the number of tones per context raises mean utility per message from 0.387 (plain glance or one tone) to 0.409; two tones give most of it, and the gain is nil for low urgency and 0.062 for high urgency (partly built in by the weights). Always repeating is barely better than one tone. A Monte Carlo check agrees with the closed form to 0.002. Tones are assumed independent and annoyance flat per tone, and every parameter is a round figure of mine: nothing is fitted or measured.
research/ambient-misread · 8 tests
How accurate an urgency label must be before an always-on assistant should act on it: naive, Bayesian and label-ignoring policies in the ambient-urgency utility model (stylised)
Abstract
In the stylised urgency model, perfect urgency knowledge is worth 0.034 utility per message. Acting naively on a noisy label stops being worse than ignoring it at about 70% accuracy with uniform errors (about 53% if errors only hit neighbouring classes), and at 70% it loses in 18 of 48 contexts. A Bayesian reading of the label never loses to ignoring it when the prior and confusion matrix are known, but a prior that over-weights high urgency drops it below ignoring the label. Every parameter is a round figure of mine: nothing is fitted or measured.
research/ambient-confest · 6 tests
How many audited messages an assistant needs to estimate a noisy urgency label's confusion matrix and prior well enough to act on it: about 50 at moderate accuracy, with smoothing helping at tiny audit sets (stylised)
Abstract
Extending the stylised urgency-label model, the assistant estimates the label’s class mix and confusion matrix from n audited messages and then acts by Bayes’ rule. With maximum-likelihood estimates, n = 10 is worse than ignoring the label (mean utility 0.363 against 0.382 at 70% label accuracy); n = 50 recovers 76% of the gain of knowing the true parameters and n = 200 recovers 92%. Add-one smoothing cuts the spread at tiny n but costs a little at high accuracy. Stylised: round-number utilities, uniform label errors, noise-free audit labels.
research/ambient-noisyaudit · 7 tests
What a noisy human auditor does to an assistant that learns its urgency label's reliability from audits: taking audit labels as truth biases the estimate and loses 8% to 69% of the label's gain at a 0.9 to 0.7 auditor, and correcting for a known auditor accuracy recovers it (stylised)
Abstract
Extending the stylised urgency-label model, the audit labels are themselves wrong. Taking them as truth biases the estimated label accuracy (0.57 instead of 0.70 with a 0.8-accuracy auditor) and does not improve with more audits: the policy loses 47% of the label’s gain over ignoring it at label accuracy 0.7, and 31% at 0.9. A 0.9 auditor costs only 8–14%. Maximum likelihood over the unseen true class, given the auditor’s accuracy, recovers the true-parameter value, but it needs that accuracy known (not identifiable from one auditor) and is no better at 50 audits with a poor auditor. Stylised; both label and auditor are made-up uniform-error matrices.
research/ambient-goldaudit · 7 tests
How many gold-labelled audits replace noisy ones when the auditor's accuracy is unknown: 25 gold audits match 1000 noisy ones read as truth, and a joint EM using 10 gold plus 200 noisy audits nearly reaches the true-parameter policy (stylised)
Abstract
Extending the stylised noisy-audit model, 200 or 1000 audits from a 0.8-accuracy auditor read as truth give mean utility 0.387–0.388 (label accuracy 0.7) and do not improve with n, while 25 gold audits already match them and 10 do not. Pooling noisy with gold audits as truth is worse than gold alone. Fitting the mix, the label matrix and the auditor’s matrix jointly by EM, with 10 gold and 200 noisy audits, reaches 0.4054 against 0.4066 for the true parameters at label accuracy 0.9 (gold alone: 0.3955). The advantage vanishes by 200 gold audits. Stylised: uniform-error matrices, one auditor accuracy, EM started at the true value, round-number utilities.
research/ambient-auditbudget · 5 tests
How to split a fixed audit budget between gold and noisy audits when gold costs 3 to 30 times more: all-noisy is worst or near-worst in all 12 cells, 25% on gold with a joint EM fit is within 0.0009 of the best, and all-gold loses when gold is expensive (stylised)
Abstract
Extending the stylised gold-audit model with a price: a gold audit costs 3, 10 or 30 noisy audits, with budgets of 200 and 1000. Spending everything on noisy audits read as truth is the worst or near-worst split in all 12 cells (0.387–0.388 at label accuracy 0.7). There is no single best share, but 25% of the budget on gold, fitted jointly with the noisy audits by EM, is within 0.0009 of the best in every cell and 50% within 0.0016. When gold is expensive, all-gold is worse (0.3843 at B = 200, cost 30). Stylised: uniform-error matrices, one auditor accuracy, round-number utilities.
research/ambient-dsaudit · 6 tests
Can the assistant learn its urgency label's reliability from two noisy auditors with no accuracy known (Dawid-Skene), and what does a wrong assumed auditor accuracy cost: over-trusting the auditor loses 31% of a 0.9 label's gain, and Dawid-Skene reaches within 0.0005 of the oracle at n = 1000 (stylised)
Abstract
Extending the noisy-audit model, a wrongly assumed accuracy for a 0.8-accuracy auditor costs 31% of a 0.9 label’s gain if the auditor is trusted fully, 17% if assumed 0.9 and 6% if assumed 0.6 (about 39–47% at either extreme for a 0.7 label). With two independent auditors on every message, Dawid–Skene EM learns all three confusion matrices with none supplied: it recovers the true-parameter value exactly in the population limit and at n = 1000 is within 0.0005 of the oracle given the same number of labels, 0.0009 to 0.0128 above taking one auditor as truth. At n = 50 with two 0.7 auditors it still falls below ignoring the label in 18% of audit sets. Stylised: uniform errors, independent auditors, round-number utilities; nothing measured on people.
research/ambient-dscorr · 4 tests
What does Dawid-Skene do when the two auditors share their errors: it over-trusts them (auditor accuracy read as 0.909 for a true 0.8 at half shared) and loses part of its edge over trusting one auditor, falling to equal it when the second auditor is a copy, never below it (stylised)
Abstract
Extending the two-auditor model, the second auditor copies the first with probability ρ (each stays 0.8 accurate). Dawid–Skene reads their agreement as correctness: the accuracy it assigns auditor 1 rises from 0.80 to 0.91 at ρ = 0.5 and 1.00 at ρ = 1. At label accuracy 0.9 its policy utility falls from 0.4066 (ρ = 0) to 0.4026 and 0.3992, where it equals taking one auditor as truth, and never goes below that in this model; an oracle told auditor 1’s accuracy is unaffected. Stylised: one shared-error model, uniform errors, round-number utilities; nothing measured on people.
research/ambient-dshard · 7 tests
What if the auditors only share errors on hard messages: Dawid-Skene's cost is much smaller than with copying (policy utility 0.4060 vs 0.4066 at 30% hard messages), reaching the naive estimator only in an extreme cell, and a hardness-aware EM recovers the classifier but cannot identify the hard share (stylised)
Abstract
A fraction h of messages is hard for both auditors (accuracy 0.5 or chance) and the rest easy, with each auditor still 0.8 accurate overall. Dawid–Skene again over-trusts the auditors (0.84 for a true 0.8 at h = 0.3, hard accuracy 0.5), but the utility cost is far smaller than with copying: at label accuracy 0.9 its policy utility is 0.4060 against 0.4066 for the true parameters. Only in the extreme cell (30% of messages at chance, the rest perfectly easy) does it fall to the naive estimator’s 0.3992, and it never goes below it. A hardness-aware EM with an informed start recovers the classifier’s matrix and the policy utility at the population level, but the hard share itself is not identified: different starts fit the same joint distribution with P(hard) from 0.24 to 0.35. Stylised, simulated results only.
research/ambient-third · 5 tests
Does a third auditor identify the hard-message share: yes at the population level (P(hard) 0.300 from informed starts), only loosely at n=1000 (sd 0.07), and in the extreme cell plain Dawid-Skene with three auditors falls below the naive estimator (0.3912 vs 0.3992) while the hardness-aware EM does not (stylised)
Abstract
Extending the hard-message model to three auditors (each 0.8 accurate overall, a fraction h of messages hard for all). A hardness-aware EM now recovers the hard share at the population level (P(hard) 0.300 from four informed starts, where two auditors left 0.24–0.35), but only loosely from samples (sd 0.07 at n = 1000) and a poor start still lands on a degenerate fit. Plain Dawid–Skene gets worse with more auditors: in the extreme cell (30% of messages at chance) it reads them as 0.85 accurate and its policy utility falls to 0.3912 at label accuracy 0.9, below the naive estimator’s 0.3992; the hardness-aware EM stays at the true-parameter 0.4066. Stylised: one shared-hardness model, uniform errors, round-number utilities; nothing measured on people.
research/ambient-multistart · 8 tests
Does a likelihood-ranked multi-start make hardness-aware Dawid-Skene robust to a poor start: at the population level yes (P(hard) 0.300, vs 0.500 and a lower likelihood from a poor single start), at finite n it matches one informed start, and where hardness is absent the hard share is not identified (stylised)
Abstract
Fitting the three-auditor hardness-aware Dawid–Skene model from 8 grid starts (none near the truth) and keeping the highest-likelihood fit recovers the hard share at the population level (P(hard) 0.300), where a single poor start stays at 0.500 with a lower likelihood. At finite n it matches one informed start (P(hard) 0.290 vs 0.290 at n = 1000, hard accuracy 1/3) and avoids the poor start’s utility loss (0.3913 vs 0.3843). With no hardness at all the hard share is not identified, and fitted shares of 0.3–0.4 appear anyway. Stylised.
research/ambient-startcount · 5 tests
How many starts does a likelihood-ranked multi-start need, and are grid starts better than random ones: one start is nearly enough with strong hardness or n = 1000, 4 starts reach the best optimum in most cells, more help only in the weak-hardness and null cells, and policy utility barely moves (stylised)
Abstract
Fitting the three-auditor hardness-aware Dawid–Skene model from 1, 2, 4, 8 or 16 nested grid starts versus the same number of random starts, keeping the highest-likelihood fit. When hardness is present and either strong or nWith strong hardness or n = 1000 one start nearly always reaches the best optimum and 4 starts reach it in every such cell;nbsp;=With strong hardness or n = 1000 one start nearly always reaches the best optimum and 4 starts reach it in every such cell;nbsp;1000, one start usually reaches the best optimum (72With strong hardness or n = 1000 one start nearly always reaches the best optimum and 4 starts reach it in every such cell;ndash;100%) and 4 starts reach it in 82With strong hardness or n = 1000 one start nearly always reaches the best optimum and 4 starts reach it in every such cell;ndash;100% of draws; with mild hardness at n = 200 and under the null, more starts keep improving the likelihood (grid hit rate 72% at K = 1 to 97% at K = 16) but mean policy utility moves by at most about 0.0002. Grid versus random is mixed. Stylised: one dependence model, 40 draws per cell, optimum judged against the best of 32 fits.
research/ambient-hlrt · 4 tests
Can a likelihood-ratio test tell no hard messages from some: chi-square cutoffs reject 35-72% of null data sets, a parametric bootstrap holds its level and detects h = 0.3 at n = 1000 but not h = 0.1 (stylised).
Abstract
Testing h = 0 in the three-auditor hardness-aware Dawid–Skene model. Under the null the likelihood-ratio statistic has median about 5 and 95th percentile about 13, so the chi-square cutoffs 3.84 and 5.99 reject 62–72% and 35–49% of null data sets. A parametric bootstrap (39 refits) rejects 7/60 and 0/60 under the null and detects h = 0.3 reliably at n = 1000 (100%) but only 37% at n = 200 with hard accuracy 0.5; h = 0.1 is undetectable. Stylised, 60 draws per cell.
research/ambient-hlrtlevel · 4 tests
Does the bootstrap likelihood-ratio test of no hard messages hold its level: 200 null draws per n give rejection 0.040-0.055 at nominal 0.05, all intervals containing 0.05 (stylised).
Abstract
A larger null-only check of the parametric-bootstrap test from ambient-hlrt (B = 39): 200 data sets with no hard messages at each of n = 100, 200 and 500. Rejection at nominal 5% is 11/200, 9/200 and 8/200 (exact 95% intervals all contain 0.05) and at 10% is 20, 16 and 19 of 200; mean p is about 0.5. The earlier 7/60 at n = 200 looks like noise. Stylised: one dependence model, level only, intervals about ±0.03.
research/ambient-hlrtmis · 5 tests
Does the no-hard-messages bootstrap test keep its level when the three auditors differ in accuracy: 100 h = 0 draws per cell give rejection 0.00-0.10 at nominal 0.05, no inflation seen (stylised).
Abstract
Data with no hard messages but three auditors of unequal accuracy (equal, 0.9/0.8/0.7, 0.95/0.8/0.65; mean 0.8), tested against a null that forces one shared auditor matrix. With 100 draws per cell at n = 200 and 500, rejection at nominal 5% is 6–10/100 at 200 and 0–6/100 at 500; five of six exact intervals contain 0.05 and the sixth (0/100) is conservative. No inflation is seen, but intervals are about ±0.05 wide. Stylised: accuracy heterogeneity only, level only.
research/ambient-bandit · 6 tests
Can an always-on assistant learn when to speak from accept/dismiss feedback alone: greedy, epsilon-greedy and Thompson online policies vs an offline oracle, mostly negative (stylised)
Abstract
A simulated assistant learns an interruption policy online (greedy, ε-greedy or Thompson-sampled logistic model) from the user’s reaction only when it speaks. With an uninformed prior none of the three learns a useful policy in 100 h; with a prior that the intent score is informative, greedy reaches 90% of an offline oracle’s rate after a median 22.5 h and about 185 dismissals, and Thompson sampling takes 32.5 h. It still earns 10.4 against the oracle’s 12.7 per hour, and after a drop in classifier quality it does far worse than frozen offline weights (0.3 vs 4.4 per hour). Stylised: independent windows, instant noise-free feedback, picked costs.
research/ambient-ips · 5 tests
Does inverse-propensity weighting fix an assistant learning from its own accept/dismiss feedback: mostly negative, exploration cost dominates (stylised)
Abstract
A stylised follow-up to ambient-bandit: the assistant speaks with a known probability (ε-greedy) and refits its model with each labelled window weighted by 1/propensity. Mostly negative: exploration itself costs more utility than the learning gains at every ε tried, weights up to 500 make the weighted fit unstable at small ε (a standard deviation across seeds of tens of utility units per hour), and the weighting did not rescue the uninformed-prior or the drift scenarios. Only at ε = 0.02 with an informed prior is the weighted model slightly closer to the oracle (12.2 vs 11.4 per hour at 100 h, oracle 12.7). Stylised: every distribution is chosen by me.
research/ambient-fullinfo · 4 tests
Does selection bias hurt an assistant learning only from windows where it spoke: yes, a lot in this world; a full-information fit reaches the oracle, weighting recovers part (stylised)
Abstract
A stylised follow-up to ambient-ips: on the same logged history, a model fitted only on windows where the assistant spoke is compared with an inverse-propensity-weighted fit and with a full-information fit that sees every window’s label. The full-information fit reaches the oracle (12.72 against 12.71 utility per hour); the biased fit loses 1.3 to 2.1 per hour with an informed prior and fails with an uninformed one, so the bias is large, contrary to a guess in ambient-ips. Weighting recovers part of the loss only at ε = 0.02 with an informed prior. The comparison also mixes bias with data volume (no label-matched control). Stylised, simulated only.
research/ambient-labelmatch · 4 tests
Is the self-selected-feedback gap bias or label volume: mostly bias (a random sample of equal size nearly reaches the oracle); clipped weights help only with an informed prior (stylised)
Abstract
A stylised follow-up to ambient-fullinfo. A control that fits on a uniformly random sample of windows, of the same size as the biased fit’s label set, gets 12.2–12.5 utility per hour against an oracle of 12.71, while the biased fit gets 10.6–11.4 with an informed prior and fails with an uninformed one: in this world the gap is mostly selection bias. Capping the inverse-propensity weights at 10 or 50 beats raw weights and the biased fit with the informed prior, but is worse than the biased fit with the uninformed prior; self-normalised weights are noisy. The control uses labels a real assistant never sees, so it locates the loss rather than recovering it. Stylised: chosen distributions, independent windows, noise-free feedback, simulated only.
research/ambient-dr · 5 tests
Does a doubly robust estimator beat IPS on an assistant's self-selected feedback: no, a stylised negative result; its pseudo-labels carry 1/ε residuals and are worse than the biased fit except with an informed prior.
Abstract
A stylised follow-up to ambient-labelmatch. Doubly robust pseudo-labels (an outcome model plus an inverse-propensity correction on the windows where the assistant spoke) are fitted on every window and compared with biased, IPS, clipped and self-normalised fits on the same logged history. A negative result: with an uninformed prior all three variants are far worse than the biased fit (utility per hour below −15 against 12.71 for the oracle) and have huge variance, and with an informed prior they are level with or above the biased fit at they match the biased fit at ε = 0.005 and 0.02 butepsilon; = 0.005 and 0.02 but are not better than clipped IPS. Simulated windows chosen by me; nothing is claimed about real users.
research/ambient-drclip · 6 tests
Does capping the inverse-propensity correction rescue doubly robust learning from an assistant's own feedback: with an informed prior yes (the collapse at ε = 0.002 goes from 0.26 to about 12 utility/h), with an uninformed prior no; stylised, simulated only.
Abstract
A stylised follow-up to ambient-dr, which left clipping untested. Capping the correction weight at 1, 5, 20 or 100 (instead of 1/ε up to 500) removes the collapse with an informed prior: at ε = 0.002 the unclipped fit scores 0.26 utility per hour and the capped ones 11.7–12.2, against 10.6 for the biased fit and 12.71 for the oracle. With an uninformed prior it does not help: capped fits that use the clipped-IPS outcome model stay between −17 and −50, and those that use the prior as outcome model score 4–8 at best, still below the random-subset control (about 12). Simulated windows chosen by me; one cap grid, 20 seeds, nothing claimed about real users.
research/ambient-crossfit · 5 tests
Does fitting the doubly robust outcome model out-of-sample (2-fold cross-fitting) help the capped correction: no, a negative result; differences are within seed noise or worse, and the unclipped collapse at ε = 0.002 persists (1.53 vs 0.26 utility/h); stylised, simulated only.
Abstract
A stylised follow-up to ambient-drclip. The outcome model behind the doubly robust pseudo-labels is fitted on the other half of the windows (2-fold cross-fitting) instead of on the same data, with the inverse-propensity correction capped at 5, 20, 100 or not at all. Negative result: with an informed prior the cross-fitted and in-sample fits are within about 0.1 utility/h of each other wherever the cap is finite, and the unclipped collapse at ε = 0.002 remains (1.53 vs 0.26 utility/h, against 12.71 for the oracle). With an uninformed prior the gaps are large but have no consistent sign (cross-fitting is better in some cells, worse in others) and both variants stay far below the biased fit. Simulated windows with costs chosen by me; no claim about a real user or device.
research/ambient-capcv · 5 tests
Can the capped doubly robust cap be chosen by 4-fold IPS validation on the log: with an informed prior it matches the best fixed cap within seed noise, with an uninformed prior it does not rescue learning; stylised, simulated only.
Abstract
A stylised follow-up to ambient-crossfit. The cap on the inverse-propensity correction is chosen from {2, 5, 10, 20, 50, 100, ∞} by 4-fold validation, scoring each candidate with an uncapped inverse-propensity estimate of its utility. With an informed prior the chosen cap matches the best fixed cap to within seed noise (12.23 against 12.19 utility/h at ε = 0.005; 11.45 against 11.84 at 0.002, with a wider spread). With an uninformed prior it does not rescue learning (between −21 and −1 utility/h with seed standard deviations of 24–51, against about 12 for a randomly sampled log), and the validation score is noisy enough that chosen caps scatter across the grid. Stylised, simulated only; 20 seeds, no significance tests.
research/ambient-drvalid · 7 tests
A doubly robust validation score does not pick the DR cap better than IPS with the same 4 folds (equal with an informed prior, mixed with the uninformed one); 8 random folds with the DR score is best in all three uninformed cells but still far below the label-matched control (stylised).
Abstract
A stylised, mostly negative follow-up to ambient-capcv. Replacing the inverse-propensity validation score with a doubly robust one, on the same 4 folds, changes nothing with an informed prior and gives mixed results with the uninformed one (seed standard deviations of 20–51 utility/h). A doubly robust score on 8 random folds is the best selector in all three uninformed cells, but remains far below a label-matched control (about 12 utility/h against −11 to 6); score, fold count and fold type changed together, so the cause is untested. Simulated, independent windows only; preliminary.
research/ambient-drablate · 7 tests
Ablating the three changes in ambient-drvalid (score, fold count, fold type): in the uninformed cells random folds help more than the DR score (IPS on 8 random folds is as good as DR on 8), no factor wins everywhere, and every selector stays far below the label-matched control (stylised, negative).
Abstract
A stylised, mostly negative ablation of ambient-drvalid: all 8 combinations of validation score (IPS or doubly robust), folds (4 or 8) and fold type (contiguous or random) choosing the doubly robust cap, 20 seeds. With an informed prior all eight are within 0.2 utility/h. With the uninformed prior, random folds are the most consistent help: at ε = 0.02 the 4-fold IPS selector gives −1.4 (std 24) against 5.6 (std 7.3) with random folds, and IPS on 8 random folds (6.3) matches DR on 8 random folds (5.9). At ε = 0.005 the DR score is worse than IPS on contiguous folds (−26.9 vs −17.7) and no selector is clearly best; seed standard deviations are 5–55. Every selector stays far below the label-matched control (about 12). Simulated, independent windows, no significance tests.
research/ambient-drrepeat · 8 tests
Repeated random fold assignments for cap selection in the ambient-assistant world (follow-up to ambient-drablate): averaging 5 assignments helps the IPS-score selector at ε = 0.02 (3.5 → 5.3 utility/h) and nothing else beyond noise; stylised, simulated only.
Abstract
A stylised, mixed follow-up to ambient-drablate: averaging the cap-selection score over 5 random fold assignments (instead of one) with an uninformed prior, 20 seeds. At ε = 0.02 it lifts the IPS-score selector from 3.5 to 5.3 utility/h and halves its seed spread; at smaller ε it changes nothing beyond noise. Single-assignment results differ from the earlier project's, showing how much one fold permutation can matter. Every selector stays far below the label-matched control (about 12). Simulated and stylised only.
research/ambient-exploreonly · 5 tests
Fitting only on the windows an assistant explored at random fixes the uninformed-prior failure of learning from its own feedback (11.99 vs −13.08 utility/h at ε = 0.02), at an exploration cost (stylised)
Abstract
A stylised follow-up to ambient-labelmatch and the reweighting and doubly robust fixes after it. The deployed learner already draws a random sample for free: the windows it spoke on only because its ε coin came up. Fitting on those windows alone, unweighted, reaches 11.99 utility/h at ε = 0.02 and 12.25 at 0.05 with an uninformed prior, against −13.08 and −71.85 for the fit on every spoken window (oracle 12.71; 20 seeds), and as a learner it earns 6.49/h online at ε = 0.005 against −2.04. Exploration costs dismissals (online utility falls at larger ε), and with an informed prior the plain fit is better at small ε. Stylised simulation only.
research/ambient-falsewake · 6 tests
How many false wakes a day does an always-on trigger get when its frame scores are correlated? Exact grid-chain rates for "m frames in a row" on AR(1) scores, checked by Monte Carlo: a threshold set by the independent-frames rule gives 613 / 4,793 / 20,993 wakes a day (not 1) for m = 2 / 3 / 5 at phi = 0.9, while a single-frame threshold is nearly unaffected. Stylised.
Abstract
A stylised new question for track 2. A trigger fires after m consecutive 10 ms frames at or above a threshold, on Gaussian AR(1) background scores. Setting the threshold so that Pm = 1 per day (independent frames) is nearly right for m = 1 (0.78 wakes a day at φ = 0.9) but wrong and unsafe for m ≥ 2: at φ = 0.9 it gives 613, 4,793 and 20,993 wakes a day for m = 2, 3, 5. Exact rates from a grid Markov chain match Monte Carlo (largest |z| 1.69), and the threshold that really gives one a day is 2.0–2.5 sd higher. The score is a model, not a real wake-word detector, and no audio was used. Stylised.
research/ambient-wakeburst · 9 tests
How bursty are an always-on trigger's false-wake counts? On AR(1) frame scores the Fano factor is near 1 (1.05–1.5), but a 5% window-to-window drift in noise scale lifts it to about 9 for m = 3 (38 at 10%); a 10 s lockout cuts it to 2.5 but also cuts the mean by a third. Stylised.
Abstract
A stylised follow-up to ambient-falsewake. On Gaussian AR(1) frame scores, wake counts per 1,000 s window are close to Poisson (Fano factor 1.05–1.5 at φ = 0.9–0.95). A window-to-window drift in the noise scale is what makes them bursty: a 5% lognormal drift lifts the Fano factor to about 9 for m = 3 while moving the mean only 4%, and 10% drift gives 38. A 10 s lockout cuts the Fano factor (9.8 to 2.5 with drift) but also cuts the mean by a third; a sub-second lockout changes little. The score is a model, not a real wake-word detector, and the drift is assumed. Stylised.
research/ambient-driftwindow · 10 tests
How does the Fano factor of false-wake counts grow with the counting window when the noise level drifts for hours? At 5% persistent drift (about 3 h correlation time) it grows from 1.9 at 100 s to about 150 at 28 h, against a flat 1.9 for independent drift. Stylised; the drift is assumed.
Abstract
A stylised follow-up to ambient-wakeburst, which drew a fresh noise level per window. With a noise scale that drifts persistently (log-scale AR(1), 5% sd) the Fano factor of false-wake counts rises with the window instead of staying flat: about 1.9 at 100 s for any correlation time, then 14, 110 and 150 at 16, 256 and 1,024 blocks (28 h) for a 2.8 h correlation time, and 620 at 28 h for a 28 h correlation time, while the mean moves only 5%. A Poisson-given-scale (Cox) model, checked against a frame-level simulation (4.7 vs 4.7 and 44 vs 44), and a closed form (mean error 2% on the busy trigger, worst 11%) explain it. The drift is assumed, not measured, and the ‘rare’ trigger is one wake per 500 s, not one a day. Stylised.
research/ambient-gatesimd · 3 tests
Do a dual-MAC energy loop and a divide-free threshold make the Cortex-M4 voice gate cheaper: 17-37% fewer instructions with the block-minimum floor, slower without it, RAM doubled; decisions unchanged (emulator counts, synthetic audio)
Abstract
A stylised follow-up to ambient-gatem4. With the block-minimum floor the SMLALD energy loop cuts executed instructions per frame by 17% at -Os and 37% at -O2, but without it the gate gets slower (64-bit ring entries), RAM doubles and code grows by about 60%. A divide-free sum-domain threshold matched the original decision on all 36,000 frames per configuration and removes the only library call; the divide itself is not counted, so its saving is unmeasured. Emulator instruction counts on synthetic audio, not cycles and not hardware.
research/ambient-gatem4 · 3 tests
What does the adaptive voice gate cost on a Cortex-M4? ambient-gatefw's C gate built with clang for Thumb-2 and run in an emulator: bit-exact, about 250 bytes of code, 3237 instructions per frame exact and 2096 with the block-10 floor, because the energy sum dominates. Instructions, not cycles; stylised.
Abstract
A stylised follow-up to ambient-gatefw. Its C gate is built with clang for Cortex-M4 (Thumb-2, soft float) and run frame by frame in an emulator on ambient-gateroc’s synthetic audio. It is bit-exact with the Python reference, takes 236–258 bytes of code at -Os, and its only library call is the 64-bit divide, once per frame. The exact gate executes 3237 instructions per 20 ms frame at -Os; the block-10 floor executes 2096, a 35% cut, because the 320-sample energy sum (about 1980 instructions) is the floor under the saving. Instructions are not cycles, the divide is not counted, and nothing ran on hardware. Stylised.
research/ambient-gatefw · 6 tests
Can a noise-adaptive voice gate fit in about 120 bytes of RAM on a pendant microcontroller? C port of ambient-gateroc's adaptive energy gate, bit-exact with the Python reference; a block-minimum noise floor cuts state from 1020 to 120 bytes and comparisons per frame from 249 to 24, moving modelled battery life by at most 0.07 h. Synthetic audio, host build only.
Abstract
A stylised follow-up to ambient-gateroc. Its adaptive energy gate (open when a 20 ms frame is at least 4× the minimum energy of the last 5 s) is ported to plain C99 and checked frame by frame against the Python reference. The exact 250-frame floor needs 1020 bytes of state and about 244 comparisons per frame. Keeping one minimum per block of 10 frames cuts that to 120 bytes and about 24 comparisons, flips 0.02–0.08% of gate decisions, and changes mean recall by at most 0.0002 and modelled battery life by at most 0.07 h (of about 82–90 h) on synthetic audio, 20 seeds × 4 rooms. Limitations: synthetic audio only; host gcc struct sizes and comparison counts, not measured cycles on an ARM core; battery hours inherit the ambient-wearable model.
research/ambient-preroll · 7 tests
How much pre-roll buffer does an energy-gated audio pendant need to stop clipping the first word? Synthetic audio: no gain in quiet rooms, in the loudest room onset recall 0.866 → 0.980 → 0.999 at 0, 5 and 25 buffered frames (0, 3.1, 15.6 KB) for 90 → 81 → 70 modelled hours; a too-deaf gate is not rescued (stylised)
Abstract
A stylised follow-up to ambient-gateroc, which did not measure onset clipping. On synthetic speech over background noise at −60 to −30 dBFS (20 seeds × 120 s per room), a noise-adaptive gate opens on the first frame in quiet rooms, so a pre-roll ring buffer buys nothing and costs battery (82.4 h to 76.9 h at 5 frames). In the loudest room it opens 2.3 frames late on average, and onset recall (first 500 ms of each utterance) goes 0.866, 0.980, 0.999 for 0, 5 and 25 buffered 20 ms frames (0, 3.1, 15.6 KB), at 90.0, 81.1 and 69.7 modelled hours. A gate set too deaf (k = 8) is not rescued: 50 frames lift onset recall only from 0.610 to 0.877. Pre-roll and hangover cost the same per frame and protect different ends of an utterance. Synthetic audio, a stylised power model, no hardware.
research/ambient-gateroc · 8 tests
Does a fixed energy-gate threshold survive different rooms on an always-on audio pendant? Synthetic audio, 20 seeds x 4 background levels: any fixed threshold that keeps speech recall >= 0.9 leaves the gate open all the time in the loudest room (57.6 h, no gating gain), while one adaptive threshold (4x a 5 s running-minimum floor) gives 82-90 h at recall 0.89-1.00 in all four; stylised, nothing measured on hardware
Abstract
A stylised follow-up to ambient-wearable. Synthetic harmonic ‘speech’ bursts (−36 to −20 dBFS) over pink-ish background noise at −60, −50, −40 and −30 dBFS plus short transients (20 seeds × 120 s per room) are passed through a Python port of the pendant’s energy gate (checked frame by frame against the compiled C), and the gate-open fraction is turned into battery hours with the same power model. Any fixed threshold that holds worst-room speech recall at 0.9 or more ((−30 dBFS or lower)minus;34 dBFS or lower on the grid I swept) stays open nearly all the time in the loudest room, giving 57.6 h, the always-streaming figure. One adaptive threshold (4× a 5 s running-minimum noise floor) gives 82–90 h at recall 0.89–1.00 in all four rooms. Within a single room the two trade-off curves are almost the same; the adaptive gate’s gain is that one setting transfers. Stylised: synthetic audio, uniform one-microphone model, assumed power parameters, nothing measured on hardware or real speech.
research/ambient-wearable · 8 tests
What limits battery life in a pendant-style always-on audio wearable: stylised nRF52840-class power model (mic floor ~170 h; speech-gated ~105 h vs 59 h always streaming) plus a 74-byte VAD gate that compiles for Cortex-M4; no hardware measured
Abstract
A stylised average-current model for an nRF52840-class pendant with a 150 mAh cell, built from a few vendor figures read on 2026-10-13 plus labelled assumptions. The always-on microphone is the floor (about 170 h alone); gating codec and radio on speech at a 30% speech fraction gives about 105 h against 59 h for always streaming. A 74-byte energy-gate routine compiles for Cortex-M4. Nothing is measured on hardware, the datasheets were not read in full and there is no numeric comparison with Omi.
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.