Should a pipeline replicate stages or skip them?

What it found

A microbatch crosses L stages held by workers that come and go. The drop probability is the exact binomial tail P(Bin(L,q)>s) with stage outage q = ρ + (1−ρ)(1−a)^r, so replicas cannot lift success above the shared-domain floor (1−ρ)^L: at L=48, ρ=10⁻³ no r reaches a 1% drop rate, but allowing one skipped stage makes r=3 enough. At a=0.9 a four-stage skip budget cuts the workers for a 10⁻³ drop rate from 240 to 96; under a per-skip quality loss κ the cost-optimal design is skip-heavy for cheap skips and moves to replication as κ grows (gain over no-skip 2.9× down to 1.08×). Independent outages, multiplicative skip penalty. Stylised.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Decentralised training systems

• How should a model be cut across devices of unequal speed?Cutting a model across heterogeneous devices: exact greedy bottleneck partition (matches DP, with memory caps), W/V ≤ T* < W/V + m·w_max/V, speed-blind splitting costs exactly v̄/v_min (2.4× at σ=0.5), device order is the residual decision (sorts no better than arbitrary, swap search within 1.1%), memory inverse to speed costs 1.7×.
• What does it cost to skip the slowest workers?Waiting for the fastest k of n workers: exact order-statistic round times (Pareto mean finite iff n−k+1 > 1/α), exact SGD time-to-accuracy with a minimum k and an interior optimum (34× over wait-for-all at s=0, 1.12× at s=32), joint step tuning, and the exact selection bias of the speed filter with Horvitz–Thompson correction.
• Does a bigger swarm learn more from shared rollouts?Rollout sharing in swarm RL as a public-goods game: Nash gives every agent a lone agent's learning level whatever N (total effort caps at (a/c−1)/θ) while efficient welfare is 4.5× at N=256; at θ=1 one agent generates; a Pigouvian matching subsidy (fraction θ(N−1)/(1+θ(N−1)) of cost) implements the optimum; junk beats honesty iff c−ε > p(τ+F).
• How stale can a shared rollout be before it is worthless?Stale off-policy rollouts in swarm RL: exact importance-weight moments (E w²=e^{δ²}, ESS fraction e^{−δ²}), closed-form truncation bias/variance verified by simulation, best cap cuts shift-estimation MSE 1.9× at δ=1 and ~750× at δ=3, and the staleness window √ln(1/ρ)/drift.
• How fast does a shared model forget a task, and does splitting it into modules help?Exact catastrophic-forgetting law for sequential training on random low-rank tasks: (r/d)(1−r/d)(ρᵏ−λᵏ) rises then vanishes, uniform module routing dilutes it but sharing wins on seen-plus-fresh loss, conflicting tasks leave a floor 2rτ²/d.
• How should independently trained modules be merged, and how much does parallel training buy?Merging independently trained modules on random low-rank tasks: exact residual 1−2amp+a²(mp+m(m−1)p²), optimal scale 1/(1+(m−1)p) (averaging stalls at (1−p)², unscaled sum diverges), harmonic 1/m decay, parallel speedup only logarithmic (8.9× at m=16, d/r=8).

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.