When should a paid verifier stop checking?

research/stopping-scores

Verifiers who choose when to stop checking under a proper score: optimal policy is a symmetric threshold (matches dynamic programming), closed-form accuracy 1/(1+((1−a)/a)^k) and cost k(2p_k−1)/(2a−1), participation κ ≥ 4c/(2a−1)², and adaptive stopping matches fixed-sample accuracy with 2–3× fewer checks.

8 tests

What it found

A verifier who checks a claim one noisy check at a time and stops when it likes, paid a proper score, follows a symmetric threshold rule (confirmed against dynamic programming over all policies). Accuracy is p_k = 1/(1+((1−a)/a)^k) and expected checks are k(2p_k−1)/(2a−1), exactly. Participation needs a score scale κ ≥ 4c/(2a−1)² under Brier (log needs about half), Brier’s threshold is ln(κ(2a−1)²/((1−a)c))/λ within one step, and adaptive stopping matches a fixed-sample verifier’s accuracy with 2–3× fewer checks. Binary state, known accuracy.

A figure from its own output

Figure produced by research/stopping-scores: checks-vs-kappa — stopping-scores/experiments/checks-vs-kappa.svg, written by the project's own experiment code.

Paper, code and results

More in Effort & peer prediction

Peer-prediction markets for decentralized ML training verificationPeer-prediction and prediction-market mechanisms for verifying decentralized ML training, with adversarial evaluation.
What does it cost to pay a verifier to check?Paying verifiers for costly checking effort with proper scoring rules: implementable efforts = concave envelope of the value curve, exact Brier rent (n−1)/(a+b), log score needs half Brier's payout spread on rare-fault priors.
What does a proper score buy when effort is private?Scoring rules as limited-liability effort contracts: shirking threshold 2c/(kκ²), first best needs a fee, otherwise the principal keeps 48–68% of surplus.
How do you pay for effort when nobody knows the truth?Peer prediction without ground truth: output agreement pays 1.0 for laziness vs 0.68 for honesty, Dasgupta–Ghosh payoff 2p(1-p)g_i g_j exact, effort as a coordination game with threshold c/(2p(1-p)κ²), and closed-form gold-check rate √(c·g_H/G).
Does agreement pay reward shared laziness?Peer prediction when lazy verifiers share a default answer: agreement pay is 2G·Cov, effort is a coordination game with threshold a*=(k/2G+s)/(pq+s), the all-lazy equilibrium always exists and pays as much as honesty when the default matches the prior, and gold checks needing more mass than the honest pay close it.
How noisy is peer-prediction pay with finitely many tasks?Finite-sample noise in multi-task peer prediction: the plug-in payment is 2× the sample covariance with an exact variance (matches enumeration), √2 less noisy than fresh penalty tasks, pays a constant reporter 0, and a lazy coin still earns ≈σ/√(2π) under limited liability; deductibles and detection/ranking sample sizes in closed form.

All 9 Effort & peer prediction projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.