How many honest runs does a slashing tolerance need?

research/conformal-tolerance

Calibrating slashing tolerances from honest runs: split-conformal false-slash law (n+1-k)/(n+1) is exact but only ~51–56% of calibration sets are ≤ α, PAC safety needs n ≥ ln δ/ln(1−α) (459 at 1%/1%), the margin is cheater hiding room (×1.26 at n=2000, Pareto-3), Gaussian plug-ins over-slash skewed drift up to 17×, pooled hardware calibration over-slashes the noisy class 4.8× and lets the quiet class hide 2.9×.

7 tests

What it found

A conformal tolerance (the k-th order statistic of n honest drift scores) has an exact, distribution-free false-slash rate (n+1−k)/(n+1) — but only 51–56% of calibration sets deliver a rate ≤ α. A PAC-safe tolerance exists iff n ≥ ln δ/ln(1−α) (459 runs at 1%/1%), and its margin is hiding room for a cheater: the threshold sits 1.26× above the true quantile at n=2000 on Pareto-3 drift. Gaussian plug-in tolerances over-slash skewed drift up to 17×; pooling calibration across mixed hardware slashes the noisy class 4.8× too often and lets the quiet class hide 2.9× more.

A figure from its own output

Figure produced by research/conformal-tolerance: inflation — conformal-tolerance/experiments/inflation.svg, written by the project's own experiment code.

Paper, code and results

More in Detection & tolerances

Should drift tolerances be per layer or on the total?Per-layer vs total drift tolerances: a tolerance per layer admits L·c(L,α)σ vs z_α√L σ on the total (6.6× at L=16), the sum test ignores how a cheat is spread while the max test catches 2.7σ in one layer but lets 16σ through spread over all, and a union test caps the hidden budget at the sum test's.
How well does Freivalds' check work in floating point?Freivalds' matrix-product check in floating point: rank-one corruption missed with probability exactly erf(t/(√2F)) (worst case for fixed norm), erf^m for m probes, Rademacher probes stuck at ½ for any size, measured float32 noise floor, and 1/erf grinding tries when the probe is hashed from the result.
How many skipped training steps hide in the loss noise?Detecting skipped training steps from the loss alone: hidden compute is the loss curve inverted at the noise band (f* = 1−(1+(e^q−1)/r)^{−1/β}), grows as reducible loss falls, seed noise sets a floor no eval size removes, simulation matches within 1.5 points, and the loss test is worth ≈4.4 step audits at best.
How much can a provider steal by serving a cheaper model?Detecting a provider that serves a cheaper model on a fraction of queries: mixture e-process (valid under peeking), delay ∝ 1/(f p² χ²), so the cheater's best response is to leak rarely and steal ∝ √N not N; closed-form audit rate and cost-optimal check rate (hK/2Nc)^{2/3}.
Can you slash while watching the data stream?Anytime-valid slashing of miscalibrated verifiers: peeking z-test falsely slashes 37% of honest verifiers (nominal 5%), an e-process holds ≤α by Ville, detection delay ≈ ln(1/α)/KL (within 12%), q-free mixture costs ×1.36.
Can two verifiers be compared while the data streams in?Anytime-valid duels between two verifiers: the optimal betting e-process on the Brier difference grows at exactly KL(q‖(rA+rB)/2), a peeking z-test falsely declares a winner 19–38% of the time at nominal 5%, a bet mixture stays ≤3.8% and costs 15–41% more decisions.

All 14 Detection & tolerances projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.