How accurate must an always-on assistant's urgency label be before acting on it beats ignoring it?

What it found

In the stylised urgency model, perfect urgency knowledge is worth 0.034 utility per message. Acting naively on a noisy label stops being worse than ignoring it at about 70% accuracy with uniform errors (about 53% if errors only hit neighbouring classes), and at 70% it loses in 18 of 48 contexts. A Bayesian reading of the label never loses to ignoring it when the prior and confusion matrix are known, but a prior that over-weights high urgency drops it below ignoring the label. Every parameter is a round figure of mine: nothing is fitted or measured.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Ambient AI assistant

• How many audited messages does an always-on assistant need before it can trust a noisy urgency label?How many audited messages an assistant needs to estimate a noisy urgency label's confusion matrix and prior well enough to act on it: about 50 at moderate accuracy, with smoothing helping at tiny audit sets (stylised)
• What does a noisy auditor do to an assistant that learns its urgency label’s reliability from audits?What a noisy human auditor does to an assistant that learns its urgency label's reliability from audits: taking audit labels as truth biases the estimate and loses 8% to 69% of the label's gain at a 0.9 to 0.7 auditor, and correcting for a known auditor accuracy recovers it (stylised)
• How many gold-labelled audits replace noisy ones when the auditor’s accuracy is unknown?How many gold-labelled audits replace noisy ones when the auditor's accuracy is unknown: 25 gold audits match 1000 noisy ones read as truth, and a joint EM using 10 gold plus 200 noisy audits nearly reaches the true-parameter policy (stylised)
• How should a fixed audit budget be split between gold-labelled and noisy audits?How to split a fixed audit budget between gold and noisy audits when gold costs 3 to 30 times more: all-noisy is worst or near-worst in all 12 cells, 25% on gold with a joint EM fit is within 0.0009 of the best, and all-gold loses when gold is expensive (stylised)
• Can an assistant learn its urgency label’s reliability from two noisy auditors, and what does a wrongly assumed auditor accuracy cost?Can the assistant learn its urgency label's reliability from two noisy auditors with no accuracy known (Dawid-Skene), and what does a wrong assumed auditor accuracy cost: over-trusting the auditor loses 31% of a 0.9 label's gain, and Dawid-Skene reaches within 0.0005 of the oracle at n = 1000 (stylised)
• What does Dawid–Skene do when the two auditors share their errors?What does Dawid-Skene do when the two auditors share their errors: it over-trusts them (auditor accuracy read as 0.909 for a true 0.8 at half shared) and loses part of its edge over trusting one auditor, falling to equal it when the second auditor is a copy, never below it (stylised)

All 46 Ambient AI assistant projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.