Can we elicit the mode of a mixture of honest and cheating drift?

research/mode-elicitation

Eliciting the modal drift with the window loss 1[|y−r|>α]: exact window-mass regret, best width α=σ (normal) / γ/√3 (Cauchy), closed-form critical width α* above which bimodal drift has one α-mode, and 0 bias against contamination where the mean moves 5ε.

7 tests

What it found

When some workers cheat, drift is a mixture and the mean, which squared loss elicits, lands where almost no task lies (at a 30% cheater 6σ away, the mean’s window holds 0.15 of the mass versus 0.48 at the mode). The mode is not elicitable, but the centre of the width-2α window with the most mass is, with the bounded loss 1[|y−r|>α]. Regret is an exact window-mass difference; the best window is α=σ for normal drift and γ/√3 for Cauchy (no mean exists); an equal bimodal mixture has one α-mode iff α exceeds a closed-form root α*; and under 10–40% contamination the sample α-mode has ~0 bias while the mean is off by 5ε, at a 3.7× RMSE cost when clean.

A figure from its own output

Figure produced by research/mode-elicitation: curvature — mode-elicitation/experiments/curvature.svg, written by the project's own experiment code.

Paper, code and results

More in Proper scoring & elicitation

Can a verifier be paid for a tolerance interval, not a point?Companion to interval-elicitation: Winkler interval score for drift-tolerance intervals: strictly proper for the α/2 and 1−α/2 quantiles, closed-form regret, optimal expected score 4sφ(z)/α, width-misreport costs whose asymmetry flips with the level, and Cauchy drift where scores have no mean but paired differences are bounded.
How should a tolerance interval be paid for?Paying for a drift tolerance interval: Winkler interval score is proper for the equal-tailed pair with exact regret (finite for Cauchy) but unbounded payment; the bounded width-plus-miss loss elicits the shortest interval (22–31% narrower on lognormal drift) at a 3–48× larger detection sample for α≥0.1.
How should a verifier's vector report be scored?Scoring a verifier's vector report: the precision score Σ⁻¹ is uniformly most detectable among quadratic Bregman scores (Cauchy–Schwarz), an isotropic score can need (1+κ)²/(4κ) times more tasks (25× at κ=100, power 0.29 vs 0.80), diagonal normalisation barely helps, estimated precision reaches 0.61 efficiency at m=2d draws.
What does a verifier's score gap cost in decisions?From a verifier's excess proper score to a certified cap on accept/audit/slash decision regret: exact calibration functions (Brier ε², log KL), linear transfer only for non-proper hinge, audit action tightens the guarantee.
Which surrogate should train a verifier that can escalate?Polyhedral surrogates for accept/reject/escalate verifiers: minimiser embeds the Bayes action, linear regret transfer with constant exactly 2d, logistic transfer ratio 2d(1-d)/δ, and a negative SGD result.
How do you pay a verifier whose report decides the job?Scoring verifiers whose report decides acceptance and whose outcome is seen only on acceptance: a truthful reserve exists iff the score centre τ_s ≥ τ_d (unique c0 = 1+τ_d²−2τ_dτ_s), plain Brier fails above τ_d = ½, recentring fixes it at rent τ³/3, exploration costs 1/ε payment variance.

All 28 Proper scoring & elicitation projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.