Can a likelihood-ratio test tell “no hard messages” from “some”, and is the chi-square cutoff usable?
research/ambient-hlrt
Can a likelihood-ratio test tell no hard messages from some: chi-square cutoffs reject 35-72% of null data sets, a parametric bootstrap holds its level and detects h = 0.3 at n = 1000 but not h = 0.1 (stylised).
What it found
Testing h = 0 in the three-auditor hardness-aware Dawid–Skene model. Under the null the likelihood-ratio statistic has median about 5 and 95th percentile about 13, so the chi-square cutoffs 3.84 and 5.99 reject 62–72% and 35–49% of null data sets. A parametric bootstrap (39 refits) rejects 7/60 and 0/60 under the null and detects h = 0.3 reliably at n = 1000 (100%) but only 37% at n = 200 with hard accuracy 0.5; h = 0.1 is undetectable. Stylised, 60 draws per cell.
A figure from its own output
Paper, code and results
More in Ambient AI assistant
- Does the bootstrap likelihood-ratio test of “no hard messages” hold its level?Does the bootstrap likelihood-ratio test of no hard messages hold its level: 200 null draws per n give rejection 0.040-0.055 at nominal 0.05, all intervals containing 0.05 (stylised).
- Does the bootstrap test of “no hard messages” stay valid when the auditors differ in accuracy?Does the no-hard-messages bootstrap test keep its level when the three auditors differ in accuracy: 100 h = 0 draws per cell give rejection 0.00-0.10 at nominal 0.05, no inflation seen (stylised).
- Can an always-on assistant learn when to speak from accept/dismiss feedback alone, and what does it cost in dismissals?Can an always-on assistant learn when to speak from accept/dismiss feedback alone: greedy, epsilon-greedy and Thompson online policies vs an offline oracle, mostly negative (stylised)
- Does inverse-propensity weighting fix an always-on assistant learning from its own accept/dismiss feedback?Does inverse-propensity weighting fix an assistant learning from its own accept/dismiss feedback: mostly negative, exploration cost dominates (stylised)
- Does selection bias actually hurt an always-on assistant that learns only from windows where it spoke?Does selection bias hurt an assistant learning only from windows where it spoke: yes, a lot in this world; a full-information fit reaches the oracle, weighting recovers part (stylised)
- Is the gap in an assistant that learns only from windows where it spoke caused by selection bias or by too few labels?Is the self-selected-feedback gap bias or label volume: mostly bias (a random sample of equal size nearly reaches the oracle); clipped weights help only with an informed prior (stylised)
All 46 Ambient AI assistant projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.