Research topic
Detection & tolerances
Statistics for catching cheats under floating-point noise: tolerances, anytime-valid tests and robust aggregation.
14 projects, each with its own page, code and tests. See them on the full research index.
research/reproducible-refereed-training · 10 tests
Measured float32 reduction-order drift, RepOps-style canonical order, Merkle-committed training traces with a teacher-forced referee; per-step tolerance is a cumulative hidden-deviation budget.
Abstract
Measured float32 drift across reduction orders (relative drift is Cauchy-tailed, state-level drift is a few ulps), a Merkle-committed training trace with a teacher-forced referee, and the finding that a per-step tolerance is a cumulative hidden-deviation budget while bitwise-reproducible operators remove it.
research/conformal-tolerance · 7 tests
Calibrating slashing tolerances from honest runs: split-conformal false-slash law (n+1-k)/(n+1) is exact but only ~51–56% of calibration sets are ≤ α, PAC safety needs n ≥ ln δ/ln(1−α) (459 at 1%/1%), the margin is cheater hiding room (×1.26 at n=2000, Pareto-3), Gaussian plug-ins over-slash skewed drift up to 17×, pooled hardware calibration over-slashes the noisy class 4.8× and lets the quiet class hide 2.9×.
Abstract
A conformal tolerance (the k-th order statistic of n honest drift scores) has an exact, distribution-free false-slash rate (n+1−k)/(n+1) — but only 51–56% of calibration sets deliver a rate ≤ α. A PAC-safe tolerance exists iff n ≥ ln δ/ln(1−α) (459 runs at 1%/1%), and its margin is hiding room for a cheater: the threshold sits 1.26× above the true quantile at n=2000 on Pareto-3 drift. Gaussian plug-in tolerances over-slash skewed drift up to 17×; pooling calibration across mixed hardware slashes the noisy class 4.8× too often and lets the quiet class hide 2.9× more.
research/layer-tolerance · 8 tests
Per-layer vs total drift tolerances: a tolerance per layer admits L·c(L,α)σ vs z_α√L σ on the total (6.6× at L=16), the sum test ignores how a cheat is spread while the max test catches 2.7σ in one layer but lets 16σ through spread over all, and a union test caps the hidden budget at the sum test's.
Abstract
Giving each of L layers its own tolerance lets a prover hide the sum of them: L·c(L,α)σ versus zα√Lσ for one test on the total, 6.6× more at L=16 and 23× at L=128. The sum test’s power depends only on the total shift; the exact (Šidák) max test’s depends on how it is spread, the prover’s best split is even (log Φ is concave), and the max test catches a 2.7σ one-layer shift half the time but lets 16σ through spread over all 16 layers. A union of both at α/2 caps the hidden budget at the sum test’s (7.8σ) while keeping one-layer sensitivity. Gaussian, independent layers, known σ.
research/freivalds-float · 7 tests
Freivalds' matrix-product check in floating point: rank-one corruption missed with probability exactly erf(t/(√2F)) (worst case for fixed norm), erf^m for m probes, Rademacher probes stuck at ½ for any size, measured float32 noise floor, and 1/erf grinding tries when the probe is hashed from the result.
Abstract
Checking a claimed matrix product by A(Br) ≈ Cr needs a tolerance t, so the error bound becomes size-dependent: a rank-one corruption of Frobenius norm F is missed by a Gaussian probe with probability exactly erf(t/(√2F)), rank one is the worst case for fixed F, and m probes give erfm (4 probes for 1% at 3t). Rademacher probes miss a two-entry corruption with probability ½ at every size. Emulated float32 noise grows ≈n0.9; a hashed probe with free retries lets a cheater grind in 1/erf tries (125 at 100t). Stylised, additive corruption.
research/compute-shirking · 14 tests
Detecting skipped training steps from the loss alone: hidden compute is the loss curve inverted at the noise band (f* = 1−(1+(e^q−1)/r)^{−1/β}), grows as reducible loss falls, seed noise sets a floor no eval size removes, simulation matches within 1.5 points, and the loss test is worth ≈4.4 step audits at best.
Abstract
If a worker paid for T steps runs only (1−f)T, the verifier’s log-loss test (null sd σ′, level α) hides exactly the compute at which the loss curve, inverted, crosses the noise band: L((1−f*)T) = L(T)eσ′(zα+zP). For a power law with a loss floor this is closed form and grows as the reducible share r falls (6% at r=1 but 64% at r=0.05 for σ′=0.02). Run-to-run seed noise is a floor that no evaluation size removes: with 2% seed noise a 5% cap is unreachable for r ≥ 0.05, while without it 104 to 4×106 eval samples suffice. On exactly solvable quadratic GD the predicted detection probability matches simulation within 1.5 points, and the loss test is worth only about 4.4 random step audits at best (0.16 once training is floor-dominated). Stylised: one statistic, normal approximation, known curve family.
research/inference-substitution · 10 tests
Detecting a provider that serves a cheaper model on a fraction of queries: mixture e-process (valid under peeking), delay ∝ 1/(f p² χ²), so the cheater's best response is to leak rarely and steal ∝ √N not N; closed-form audit rate and cost-optimal check rate (hK/2Nc)^{2/3}.
Abstract
A decentralised inference provider substitutes a cheaper model on a random fraction p of queries; a verifier re-scores a fraction f under an anytime-valid mixture e-process. Detection delay scales as 1/(f p² χ²), so the profitable attack is a slow leak: undetected savings grow as √N rather than N (measured ×7.75 for ×64 queries). Closed-form audit rate to cap theft, cost-optimal check rate (hK/2Nc)2/3, and a substitute close enough to the claimed model that even checking every response cannot cap the theft. Seeded Monte Carlo. Stylised.
research/sequential-slashing · 6 tests
Anytime-valid slashing of miscalibrated verifiers: peeking z-test falsely slashes 37% of honest verifiers (nominal 5%), an e-process holds ≤α by Ville, detection delay ≈ ln(1/α)/KL (within 12%), q-free mixture costs ×1.36.
Abstract
Anytime-valid slashing of miscalibrated verifiers with e-processes: repeatedly re-running a z-test falsely slashes 37% of honest verifiers at nominal 5%, the likelihood-ratio e-process stays under α by Ville’s inequality, detection delay tracks ln(1/α)/KL within 12% (logarithmic in 1/α), and not knowing the cheating rate costs a factor 1.36. Stylised.
research/forecast-duel · 7 tests
Anytime-valid duels between two verifiers: the optimal betting e-process on the Brier difference grows at exactly KL(q‖(rA+rB)/2), a peeking z-test falsely declares a winner 19–38% of the time at nominal 5%, a bet mixture stays ≤3.8% and costs 15–41% more decisions.
Abstract
For fixed reports rA, rB and binary outcomes with frequency q, the Brier difference is a two-point gamble and A is better exactly when q is on A’s side of the midpoint π=(rA+rB)/2. Betting a fraction λ of wealth on it gives an anytime-valid e-process whose best bet grows at exactly KL(q‖π), so the expected decision time is ln(1/α)/KL. A z-test checked after every task falsely declares a winner in 19–38% of null runs at nominal 5%; a uniform mixture over bets stays at or below 3.8% and costs 15–41% extra tasks; a bet twice too large has negative growth. Fixed reports, i.i.d. outcomes. Stylised.
research/verifier-league · 11 tests
Anytime-valid best-of-n verifier identification: delay set by the rival whose report midpoint sits nearest q (the best opposite-side rival, not the runner-up), mixture within 1.35–1.60× of oracle, elimination 4–5× cheaper, peeking declares a false best in 59% of ties vs 1.6%.
Abstract
Under Brier loss the best verifier is the one whose report is closest to the outcome frequency q, and each pair has an anytime-valid betting e-process growing at KL(q‖midpoint). Certifying the leader means beating every rival, so the delay is ln(1/α) over the smallest such rate, which belongs to the rival whose midpoint with the leader is nearest q: the best rival on the opposite side of q, not the runner-up (rate 0.0026 against 0.0086 in a six-verifier league). A bet mixture certifies within 1.35–1.60× of the oracle prediction; successive elimination cuts verifier evaluations 4cuts evaluation cost 2–4× at n=8–12ndash;5× at n=8–12; a peeking z-test declares a false strict best in 59% of exact-tie runs against 1.6% for the e-process. Stylised: fixed reports, i.i.d. tasks.
research/robust-aggregation · 7 tests
Byzantine-robust aggregation of verifier reports: mean bias βB unbounded, median bias σΦ⁻¹(1/(2(1−β))) independent of B, trimmed-mean closed form for τ≥β verified by simulation.
Abstract
Mean, median and trimmed-mean aggregation of verifier reports under a Byzantine fraction β: mean bias βB is unbounded, median bias σΦ−1(1/(2(1−β))) is independent of B and diverges at β=½, trimmed-mean closed form for τ≥β matches simulation, and trimming below β fails. Stylised.
research/forgetting-audit · 7 tests
Auditing skipped or partial continual-learning training from old-task losses: exact loss law (ρ^{t−1}[(1−2m₁+m₂)(r/d)λᵏ+(r/d)ρ(ρᵏ−λᵏ)]), a skipped task is ≥ d/(d−r)× worse but only clearly so inside a lag window, detecting a skipper takes ~0.7/s² jobs at any lag, a 10% step shortfall needs ~9600.
Abstract
Extends the forgetting law to a trainer that skips each task with probability s or takes a step fraction a: the final-checkpoint loss on the task at position t is exactly ρt−1[(1−2m1+m2)(r/d)λk + (r/d)ρ(ρk−λk)], matching exact quadrature and simulation. A skipped task is at least d/(d−r) times worse than a trained one at every lag but clearly so only in a window (ratio ≥2 for lags ≤6.7 at d=16, r=2). Catching a skipping trainer is lag-free: about 17–20 jobs for 20% skipping at any audit lag, scaling as 0.7/s2 and falling with job length, while a 10% step shortfall needs about 9,600 jobs. Stylised: linear regression, Haar-random tasks, known initial error.
research/interference-twin · 10 tests
Interference-free or mean-noise-rise radar twin vs intermittent mutual radar interference between robots: exact closed-form per-frame Pd, real Pd plateaus at 0.656 (no-hit probability), mean-rise twin 9% optimistic at Pd ≥ 0.9 yet badly pessimistic for hit-tolerant missions, geometry knowledge helps 9-of-10 missions (Brier 0.089 vs 0.101). Stylised.
Abstract
Mutual radar interference between robots, with heavy-tailed c/d2 interferer power hitting each frame intermittently. Per-frame detection probability has a closed form (matches Monte Carlo within 0.003); real Pd plateaus at the probability that no interferer hits the frame (0.656) rather than shrinking in range. A mean-noise-rise twin is 9% optimistic at Pd ≥ 0.9 (63.7 vs 58.4 m) but badly pessimistic for missions that tolerate hit frames (Brier 0.690 vs 0.003 for 3-of-10); for 9-of-10 missions knowing the interferer geometry helps (0.089 vs 0.101). Stylised: simulated interference, illustrative parameters, no radar data.
research/infra-leakpdd · 5 tests
Net3 leak signals under demand-driven and pressure-driven demand (WNTR): identical at 1 to 1.25 times base demand and nearly so at 1.5 times; beyond that both models fail because the pumps cannot meet demand (simulated; one network).
Abstract
WNTR simulates EPANET’s Net3 for 72 hours at demand multipliers 1 to 3 under the default demand-driven model and the pressure-driven model (required pressure 15 m, 10 and 25 m swept), with a leak of 2 % of mean demand at 30 seeded junctions read at five sensors. At 1 and 1.25 times base demand the two models agree to the three decimals printed; at 1.5 times the median sensor drop is 0.798 m against 0.792 m and 29 of 30 locations are seen by both. At 2 and 3 times the pumps run past their rated flow, the demand-driven run has 60–91 % of junction-hours below 0 m (down to −458 m) and the leak passes no flow, while the pressure-driven run serves 86 % and 62 % of demand and still sits below the required pressure over half the time, so neither is believable there. Hence the conclusions of the earlier detector study, drawn at base demand, do not depend on the demand model. Limits: one textbook network whose low-pressure nodes are mostly zero-demand, noise-free sensor response only (no detectors or false-alarm budget), assumed required pressure, 30 locations and one leak size.
research/infra-waterloss · 6 tests
Water-leak detection from pressure residuals on EPANET's Net3 (WNTR): at a 5% weekly false-alarm budget, 5 sensors and 10% demand error, CUSUM finds all leaks the sensors can see at 4% of demand but 8% at 1% (3% of all leaks); demand error, not sensor noise or detector choice, sets the limit (simulated leaks; BattLeDIM L-Town was unreachable).
Abstract
WNTR simulates one week of hourly pressures on EPANET’s Net3 (92 junctions) with leaks of 0.25–4 % of mean demand (1.7–28 L/s) opened at hour 96 at 60 random junctions, against a nominal model whose demands are wrong (node error sd 5–20 %, system-wide AR(1) error sd 1–5 %) and sensors with noise 0.05–1 m. A moving-average threshold, CUSUM and PELT (changepoint) each get a threshold calibrated on 120 leak-free weeks to at most 5 % of weeks with any alarm, with and without a regression on a noisy inflow meter. With 5 greedily placed sensors and base demand error, the best detector (flow-adjusted CUSUM) finds 100 % of the leaks the sensors can see (28 % of locations) at 4 % of demand but only 8 % at 1 % and 85 % at 2 %, and across all leak locations 3 % at 1 % and 38 % at 4 %. Demand error moves the result far more than sensor noise (covered 2 % leaks: 92 % at low demand error, 85 % at base, 8 % at high), and averaging 15 sensors instead of 5 hurts. PELT did not beat CUSUM on 2 % leaks in any condition at the same false-alarm budget. Limits: the BattLeDIM L-Town network and its leak schedule could not be downloaded here, so there is no comparison with the competition’s methods; Net3 is a textbook network; leaks are simulated, demand error and inflow-meter noise (0.5 %) are assumed and swept, not measured; 60 locations per size, so rates carry binomial error of up to ±0.12.
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.