How often should preemptible workers checkpoint?

What it found

Exact checkpoint interval under Poisson churn: w* = (1+W₀(−e^−1−λC))/λ, independent of restart cost, with Young/Daly overshooting by 5–25%; intervals within 2× cost under 3.5%; a synchronous job on n workers restarts at rate nλ so efficiency falls to 50% near n≈700, and admitting flaky workers cut throughput five-fold. Stylised.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Decentralised training systems

• Should a pipeline replicate stages or skip them?Replicate or skip? pipeline stages on unreliable workers: exact binomial drop law, replicas cannot beat the shared-domain floor (1−ρ)^L, a 4-stage skip budget halves workers for a 1e-3 drop rate (240 → 96), cost-optimal design shifts from skipping to replication as skips get costly.
• How should a model be cut across devices of unequal speed?Cutting a model across heterogeneous devices: exact greedy bottleneck partition (matches DP, with memory caps), W/V ≤ T* < W/V + m·w_max/V, speed-blind splitting costs exactly v̄/v_min (2.4× at σ=0.5), device order is the residual decision (sorts no better than arbitrary, swap search within 1.1%), memory inverse to speed costs 1.7×.
• What does it cost to skip the slowest workers?Waiting for the fastest k of n workers: exact order-statistic round times (Pareto mean finite iff n−k+1 > 1/α), exact SGD time-to-accuracy with a minimum k and an interior optimum (34× over wait-for-all at s=0, 1.12× at s=32), joint step tuning, and the exact selection bias of the speed filter with Horvitz–Thompson correction.
• Does a bigger swarm learn more from shared rollouts?Rollout sharing in swarm RL as a public-goods game: Nash gives every agent a lone agent's learning level whatever N (total effort caps at (a/c−1)/θ) while efficient welfare is 4.5× at N=256; at θ=1 one agent generates; a Pigouvian matching subsidy (fraction θ(N−1)/(1+θ(N−1)) of cost) implements the optimum; junk beats honesty iff c−ε > p(τ+F).
• How stale can a shared rollout be before it is worthless?Stale off-policy rollouts in swarm RL: exact importance-weight moments (E w²=e^{δ²}, ESS fraction e^{−δ²}), closed-form truncation bias/variance verified by simulation, best cap cuts shift-estimation MSE 1.9× at δ=1 and ~750× at δ=3, and the staleness window √ln(1/ρ)/drift.
• How fast does a shared model forget a task, and does splitting it into modules help?Exact catastrophic-forgetting law for sequential training on random low-rank tasks: (r/d)(1−r/d)(ρᵏ−λᵏ) rises then vanishes, uniform module routing dilutes it but sharing wins on seen-plus-fresh loss, conflicting tasks leave a floor 2rτ²/d.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.