When most verifiers miss a fault, who should be believed?

What it found

If a fault is flagged by fewer than half of the verifiers (0.45 detection, 0.05 false alarms), majority vote is wrong on faulty jobs with probability 0.84 at n=101 and 0.96 at n=301. The surprisingly-popular rule (verdict plus predicted verdict fraction) reduces for truthful Bayesian verifiers to an exact fraction threshold θ = v/(1−u+v) and errs 2.1×10⁻⁵ on faulty jobs at n=101, needing 47 votes for 10⁻³ total error against 36 for a Bayes rule that knows the detector. A wrong believed prior wrecks finite-n accuracy (0.51 error), a wrong believed detector quality breaks the limit, and Byzantines who vote clean and predict “everyone flags” break it at exactly 13.6%. Stylised, truthful reports assumed.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Effort & peer prediction

• Why should the first verifiers be sealed?Sequential verifiers who see earlier verdicts herd: wrong-cascade probability exactly (1−a)²/(a²+(1−a)²), cascade after 2/(a²+(1−a)²) signals, accuracy capped at 0.845 (a=0.7) versus ≈1 for an independent majority, and a sealed first batch of ≥3 lifts the cap (0.875 at m=3, 0.949 at m=11); checked against Bayesian agents.
• When should a paid verifier stop checking?Verifiers who choose when to stop checking under a proper score: optimal policy is a symmetric threshold (matches dynamic programming), closed-form accuracy 1/(1+((1−a)/a)^k) and cost k(2p_k−1)/(2a−1), participation κ ≥ 4c/(2a−1)², and adaptive stopping matches fixed-sample accuracy with 2–3× fewer checks.
• Peer-prediction markets for decentralized ML training verificationPeer-prediction and prediction-market mechanisms for verifying decentralized ML training, with adversarial evaluation.
• What does it cost to pay a verifier to check?Paying verifiers for costly checking effort with proper scoring rules: implementable efforts = concave envelope of the value curve, exact Brier rent (n−1)/(a+b), log score needs half Brier's payout spread on rare-fault priors.
• What does a proper score buy when effort is private?Scoring rules as limited-liability effort contracts: shirking threshold 2c/(kκ²), first best needs a fee, otherwise the principal keeps 48–68% of surplus.
• How do you pay for effort when nobody knows the truth?Peer prediction without ground truth: output agreement pays 1.0 for laziness vs 0.68 for honesty, Dasgupta–Ghosh payoff 2p(1-p)g_i g_j exact, effort as a coordination game with threshold c/(2p(1-p)κ²), and closed-form gold-check rate √(c·g_H/G).

All 9 Effort & peer prediction projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.