Does the bootstrap test of “no hard messages” stay valid when the auditors differ in accuracy?

What it found

Data with no hard messages but three auditors of unequal accuracy (equal, 0.9/0.8/0.7, 0.95/0.8/0.65; mean 0.8), tested against a null that forces one shared auditor matrix. With 100 draws per cell at n = 200 and 500, rejection at nominal 5% is 6–10/100 at 200 and 0–6/100 at 500; five of six exact intervals contain 0.05 and the sixth (0/100) is conservative. No inflation is seen, but intervals are about ±0.05 wide. Stylised: accuracy heterogeneity only, level only.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Ambient AI assistant

• Can an always-on assistant learn when to speak from accept/dismiss feedback alone, and what does it cost in dismissals?Can an always-on assistant learn when to speak from accept/dismiss feedback alone: greedy, epsilon-greedy and Thompson online policies vs an offline oracle, mostly negative (stylised)
• Does inverse-propensity weighting fix an always-on assistant learning from its own accept/dismiss feedback?Does inverse-propensity weighting fix an assistant learning from its own accept/dismiss feedback: mostly negative, exploration cost dominates (stylised)
• Does selection bias actually hurt an always-on assistant that learns only from windows where it spoke?Does selection bias hurt an assistant learning only from windows where it spoke: yes, a lot in this world; a full-information fit reaches the oracle, weighting recovers part (stylised)
• Is the gap in an assistant that learns only from windows where it spoke caused by selection bias or by too few labels?Is the self-selected-feedback gap bias or label volume: mostly bias (a random sample of equal size nearly reaches the oracle); clipped weights help only with an informed prior (stylised)
• Does a doubly robust estimator fix learning from an assistant’s own self-selected feedback?Does a doubly robust estimator beat IPS on an assistant's self-selected feedback: no, a stylised negative result; its pseudo-labels carry 1/ε residuals and are worse than the biased fit except with an informed prior.
• Does capping the inverse-propensity correction rescue doubly robust learning from an assistant’s own feedback?Does capping the inverse-propensity correction rescue doubly robust learning from an assistant's own feedback: with an informed prior yes (the collapse at ε = 0.002 goes from 0.26 to about 12 utility/h), with an uninformed prior no; stylised, simulated only.

All 46 Ambient AI assistant projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.