Why should the first verifiers be sealed?

What it found

Verifiers who are paid for being right and see earlier verdicts herd: once two more revealed signals point one way than the other, everyone copies. With signal accuracy a the cascade is wrong with probability exactly (1−a)²/(a²+(1−a)²) (0.155 at a=0.7) and forms after 2/(a²+(1−a)²) revealed signals (3.4), so accuracy is capped at 0.845 however many verifiers join, versus ≈1 for an independent majority of 101. A sealed first batch of m ≥ 3 lifts the cap (0.875 at m=3, 0.949 at 11) and beats a majority of the same paid size. Recursion checked against fully Bayesian simulated agents. Stylised, symmetric signals.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Effort & peer prediction

• When should a paid verifier stop checking?Verifiers who choose when to stop checking under a proper score: optimal policy is a symmetric threshold (matches dynamic programming), closed-form accuracy 1/(1+((1−a)/a)^k) and cost k(2p_k−1)/(2a−1), participation κ ≥ 4c/(2a−1)², and adaptive stopping matches fixed-sample accuracy with 2–3× fewer checks.
• Peer-prediction markets for decentralized ML training verificationPeer-prediction and prediction-market mechanisms for verifying decentralized ML training, with adversarial evaluation.
• What does it cost to pay a verifier to check?Paying verifiers for costly checking effort with proper scoring rules: implementable efforts = concave envelope of the value curve, exact Brier rent (n−1)/(a+b), log score needs half Brier's payout spread on rare-fault priors.
• What does a proper score buy when effort is private?Scoring rules as limited-liability effort contracts: shirking threshold 2c/(kκ²), first best needs a fee, otherwise the principal keeps 48–68% of surplus.
• How do you pay for effort when nobody knows the truth?Peer prediction without ground truth: output agreement pays 1.0 for laziness vs 0.68 for honesty, Dasgupta–Ghosh payoff 2p(1-p)g_i g_j exact, effort as a coordination game with threshold c/(2p(1-p)κ²), and closed-form gold-check rate √(c·g_H/G).
• Does agreement pay reward shared laziness?Peer prediction when lazy verifiers share a default answer: agreement pay is 2G·Cov, effort is a coordination game with threshold a*=(k/2G+s)/(pq+s), the all-lazy equilibrium always exists and pays as much as honesty when the default matches the prior, and gold checks needing more mass than the honest pay close it.

All 9 Effort & peer prediction projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.