Effort & peer prediction

Proper scoring rules, Peer Truth Serum, Correlated Agreement and LMSR aggregation, with eight follow-ups on mechanism-aware adversaries (trust farming, whitewashing, closed-loop attackers) and defences. Pure-Python, dependency-free.

Effort elicitation with proper scoring rules in an exact beta-binomial model: implementable efforts are the concave envelope of the value curve, Brier information rent is exactly (n−1)/(a+b) times effort cost, and the log score needs half Brier's payout spread on rare-fault priors. Stylised.

Scoring-rule payments as moral-hazard contracts for costly inference: a closed-form shirking threshold (matched to 1%), first-best effort at scale equal to marginal value with a fee, and under limited liability only 48–65% of first-best surplus, with log score cheaper than Brier at high effort. Stylised.

Peer prediction for unverifiable labelling and judging work: output agreement pays 1.0 for always answering the majority label versus 0.68 for honesty, the Dasgupta–Ghosh payoff is exactly 2p(1−p)g_ig_j, effort becomes a coordination game with a closed-form threshold (12.8× higher for rare-positive tasks), and a cost-optimal gold-check rate √(c·g_H/G) removes the shirking equilibrium. Stylised.

Multi-task peer prediction pays 2G·Cov of two reports. If lazy verifiers copy a shared per-task default (the same public model’s guess) pay becomes 2G[a_ia_jpq + (1−a_i)(1−a_j)d(1−d)], effort is a coordination game with threshold a* = (k/2G + s)/(pq + s), and the all-lazy profile is always an equilibrium paying 2G·d(1−d): equal to honest pay when the default matches the prior, 1.19× it at p=0.3, d=0.5. Closing it with gold checks needs payment mass (2Gs+k)/(1−acc_D), larger than the honest pay itself. Stylised.

The plug-in multi-task payment (agreement minus the agreement implied by empirical frequencies) is exactly twice the sample covariance of two verifiers’ reports, so its variance is available in closed form and matches full enumeration; it is √2 less noisy than drawing fresh penalty tasks and pays a constant reporter exactly 0. Under limited liability a verifier who does no work still collects a noise rent ≈ σ/√(2π) (0.020 against 0.208 honest at n=100). A deductible removes it in closed form, but killing 99% of the rent costs the honest verifier 54% of its pay. Detection needs 56–329 tasks for g=0.7–0.3 (simulated power 0.94–0.95), and ranking two close verifiers takes ~870. Binary conditionally independent reports. Stylised.

If a fault is flagged by fewer than half of the verifiers (0.45 detection, 0.05 false alarms), majority vote is wrong on faulty jobs with probability 0.84 at n=101 and 0.96 at n=301. The surprisingly-popular rule (verdict plus predicted verdict fraction) reduces for truthful Bayesian verifiers to an exact fraction threshold θ = v/(1−u+v) and errs 2.1×10⁻⁵ on faulty jobs at n=101, needing 47 votes for 10⁻³ total error against 36 for a Bayes rule that knows the detector. A wrong believed prior wrecks finite-n accuracy (0.51 error), a wrong believed detector quality breaks the limit, and Byzantines who vote clean and predict “everyone flags” break it at exactly 13.6%. Stylised, truthful reports assumed.

Verifiers who are paid for being right and see earlier verdicts herd: once two more revealed signals point one way than the other, everyone copies. With signal accuracy a the cascade is wrong with probability exactly (1−a)²/(a²+(1−a)²) (0.155 at a=0.7) and forms after 2/(a²+(1−a)²) revealed signals (3.4), so accuracy is capped at 0.845 however many verifiers join, versus ≈1 for an independent majority of 101. A sealed first batch of m ≥ 3 lifts the cap (0.875 at m=3, 0.949 at 11) and beats a majority of the same paid size. Recursion checked against fully Bayesian simulated agents. Stylised, symmetric signals.

A verifier who checks a claim one noisy check at a time and stops when it likes, paid a proper score, follows a symmetric threshold rule (confirmed against dynamic programming over all policies). Accuracy is p_k = 1/(1+((1−a)/a)^k) and expected checks are k(2p_k−1)/(2a−1), exactly. Participation needs a score scale κ ≥ 4c/(2a−1)² under Brier (log needs about half), Brier’s threshold is ln(κ(2a−1)²/((1−a)c))/λ within one step, and adaptive stopping matches a fixed-sample verifier’s accuracy with 2–3× fewer checks. Binary state, known accuracy.