Does a doubly robust validation score choose the doubly robust cap better than an IPS score?
research/ambient-drvalid
A doubly robust validation score does not pick the DR cap better than IPS with the same 4 folds (equal with an informed prior, mixed with the uninformed one); 8 random folds with the DR score is best in all three uninformed cells but still far below the label-matched control (stylised).
What it found
A stylised, mostly negative follow-up to ambient-capcv. Replacing the inverse-propensity validation score with a doubly robust one, on the same 4 folds, changes nothing with an informed prior and gives mixed results with the uninformed one (seed standard deviations of 20–51 utility/h). A doubly robust score on 8 random folds is the best selector in all three uninformed cells, but remains far below a label-matched control (about 12 utility/h against −11 to 6); score, fold count and fold type changed together, so the cause is untested. Simulated, independent windows only; preliminary.
A figure from its own output
Paper, code and results
More in Ambient AI assistant
- Which of score, fold count and fold type made cap selection better in ambient-drvalid?Ablating the three changes in
ambient-drvalid(score, fold count, fold type): in the uninformed cells random folds help more than the DR score (IPS on 8 random folds is as good as DR on 8), no factor wins everywhere, and every selector stays far below the label-matched control (stylised, negative). - Do repeated random fold assignments stabilise validation-based cap selection?Repeated random fold assignments for cap selection in the ambient-assistant world (follow-up to
ambient-drablate): averaging 5 assignments helps the IPS-score selector at ε = 0.02 (3.5 → 5.3 utility/h) and nothing else beyond noise; stylised, simulated only. - Does fitting only on the windows an assistant explored at random fix learning from its own feedback?Fitting only on the windows an assistant explored at random fixes the uninformed-prior failure of learning from its own feedback (11.99 vs −13.08 utility/h at ε = 0.02), at an exploration cost (stylised)
- How many false wakes a day does an always-on trigger get when its frame scores are correlated, and what does requiring m frames in a row buy?How many false wakes a day does an always-on trigger get when its frame scores are correlated? Exact grid-chain rates for "m frames in a row" on AR(1) scores, checked by Monte Carlo: a threshold set by the independent-frames rule gives 613 / 4,793 / 20,993 wakes a day (not 1) for m = 2 / 3 / 5 at phi = 0.9, while a single-frame threshold is nearly unaffected. Stylised.
- How bursty are an always-on trigger's false-wake counts, and does a lockout help?How bursty are an always-on trigger's false-wake counts? On AR(1) frame scores the Fano factor is near 1 (1.05–1.5), but a 5% window-to-window drift in noise scale lifts it to about 9 for m = 3 (38 at 10%); a 10 s lockout cuts it to 2.5 but also cuts the mean by a third. Stylised.
- How does the burstiness of an always-on trigger's false-wake counts grow with the counting window when the noise level drifts for hours?How does the Fano factor of false-wake counts grow with the counting window when the noise level drifts for hours? At 5% persistent drift (about 3 h correlation time) it grows from 1.9 at 100 s to about 150 at 28 h, against a flat 1.9 for independent drift. Stylised; the drift is assumed.
All 46 Ambient AI assistant projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.