How should a model be cut across devices of unequal speed?

research/hetero-pipeline

Cutting a model across heterogeneous devices: exact greedy bottleneck partition (matches DP, with memory caps), W/V ≤ T* < W/V + m·w_max/V, speed-blind splitting costs exactly v̄/v_min (2.4× at σ=0.5), device order is the residual decision (sorts no better than arbitrary, swap search within 1.1%), memory inverse to speed costs 1.7×.

8 tests

What it found

For a fixed pipeline order, greedy filling is an exact test of a target stage time (agreeing with an independent dynamic program, with and without memory caps), and every order satisfies W/V ≤ T* < W/V + m w_max/V. A speed-blind equal split costs exactly v̄/v_min times the lower bound (2.40× at log-speed spread 0.5, 8.25× at 1.0) while speed-aware cutting stays within 1.4% at 240 layers. What is left is the device order: at 14 layers and 6 devices the worst order is 1.34× the best, fast-first and slow-first sorts do no better than an arbitrary order (all about 1.16× optimal), and pairwise-swap search gets within 1.1%; the penalty fades to 1.03× at 224 layers. Memory that shrinks as speed grows inflates the bottleneck 1.7×. No communication cost, static speeds.

A figure from its own output

Figure produced by research/hetero-pipeline: price_of_heterogeneity — hetero-pipeline/experiments/price_of_heterogeneity.svg, written by the project's own experiment code.

Paper, code and results

More in Decentralised training systems

What does it cost to skip the slowest workers?Waiting for the fastest k of n workers: exact order-statistic round times (Pareto mean finite iff n−k+1 > 1/α), exact SGD time-to-accuracy with a minimum k and an interior optimum (34× over wait-for-all at s=0, 1.12× at s=32), joint step tuning, and the exact selection bias of the speed filter with Horvitz–Thompson correction.
Does a bigger swarm learn more from shared rollouts?Rollout sharing in swarm RL as a public-goods game: Nash gives every agent a lone agent's learning level whatever N (total effort caps at (a/c−1)/θ) while efficient welfare is 4.5× at N=256; at θ=1 one agent generates; a Pigouvian matching subsidy (fraction θ(N−1)/(1+θ(N−1)) of cost) implements the optimum; junk beats honesty iff c−ε > p(τ+F).
How stale can a shared rollout be before it is worthless?Stale off-policy rollouts in swarm RL: exact importance-weight moments (E w²=e^{δ²}, ESS fraction e^{−δ²}), closed-form truncation bias/variance verified by simulation, best cap cuts shift-estimation MSE 1.9× at δ=1 and ~750× at δ=3, and the staleness window √ln(1/ρ)/drift.
How fast does a shared model forget a task, and does splitting it into modules help?Exact catastrophic-forgetting law for sequential training on random low-rank tasks: (r/d)(1−r/d)(ρᵏ−λᵏ) rises then vanishes, uniform module routing dilutes it but sharing wins on seen-plus-fresh loss, conflicting tasks leave a floor 2rτ²/d.
How should independently trained modules be merged, and how much does parallel training buy?Merging independently trained modules on random low-rank tasks: exact residual 1−2amp+a²(mp+m(m−1)p²), optimal scale 1/(1+(m−1)p) (averaging stalls at (1−p)², unscaled sum diverges), harmonic 1/m decay, parallel speedup only logarithmic (8.9× at m=16, d/r=8).
When should a continual learner replay an old task, and does a fixed schedule beat random replay?Exact expected forgetting under replay schedules in sequential training on random low-rank tasks: 2×2 matrix recursion matches simulation, one replay is best at the forgetting-peak lag yet removes only 25–42%, a fixed period beats random replay only below ~1.2–3 peak lags, halving forgetting needs rate 0.25 (d=12, r=3) to 0.015 (d=128, r=1).

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.