How noisy is peer-prediction pay with finitely many tasks?

What it found

The plug-in multi-task payment (agreement minus the agreement implied by empirical frequencies) is exactly twice the sample covariance of two verifiers’ reports, so its variance is available in closed form and matches full enumeration; it is √2 less noisy than drawing fresh penalty tasks and pays a constant reporter exactly 0. Under limited liability a verifier who does no work still collects a noise rent ≈ σ/√(2π) (0.020 against 0.208 honest at n=100). A deductible removes it in closed form, but killing 99% of the rent costs the honest verifier 54% of its pay. Detection needs 56–329 tasks for g=0.7–0.3 (simulated power 0.94–0.95), and ranking two close verifiers takes ~870. Binary conditionally independent reports. Stylised.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Effort & peer prediction

• When most verifiers miss a fault, who should be believed?Surprisingly-popular verdicts for rare-but-diagnostic signals: majority errs 0.84 on faulty jobs at n=101 when detection is 0.45, SP is an exact fraction threshold v/(1−u+v) (2.1×10⁻⁵ error, 47 votes for 10⁻³ vs Bayes 36), a wrong believed prior wrecks finite-n accuracy, a wrong believed detector breaks the limit, Byzantine breakdown at exactly 13.6%.
• Why should the first verifiers be sealed?Sequential verifiers who see earlier verdicts herd: wrong-cascade probability exactly (1−a)²/(a²+(1−a)²), cascade after 2/(a²+(1−a)²) signals, accuracy capped at 0.845 (a=0.7) versus ≈1 for an independent majority, and a sealed first batch of ≥3 lifts the cap (0.875 at m=3, 0.949 at m=11); checked against Bayesian agents.
• When should a paid verifier stop checking?Verifiers who choose when to stop checking under a proper score: optimal policy is a symmetric threshold (matches dynamic programming), closed-form accuracy 1/(1+((1−a)/a)^k) and cost k(2p_k−1)/(2a−1), participation κ ≥ 4c/(2a−1)², and adaptive stopping matches fixed-sample accuracy with 2–3× fewer checks.
• Peer-prediction markets for decentralized ML training verificationPeer-prediction and prediction-market mechanisms for verifying decentralized ML training, with adversarial evaluation.
• What does it cost to pay a verifier to check?Paying verifiers for costly checking effort with proper scoring rules: implementable efforts = concave envelope of the value curve, exact Brier rent (n−1)/(a+b), log score needs half Brier's payout spread on rare-fault priors.
• What does a proper score buy when effort is private?Scoring rules as limited-liability effort contracts: shirking threshold 2c/(kκ²), first best needs a fee, otherwise the principal keeps 48–68% of surplus.

All 9 Effort & peer prediction projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.