In the verifiable training mechanisms programme a worker commits to a checkpoint every s steps and a verifier replays k sampled segments. Two results from this week fit together.
- Replaying in fp16 drifts from the worker's float32 weights, and that drift stayed below the smallest step in its segment up to
s = 20(worst ratio 0.48 in 600 segments per cell) but not ats = 40(2.59; 1% of segments). - The chance of catching a one-step stall depends only on the replay budget
B = k sover training lengthT, not ons(simulated: 0.0321 to 0.0345 against 20/600 = 0.0333).
So s only sets overhead, and with K the spacing at which checkpoints are stored, the expected replay cost is B((K/s - 1)/2 + 1) steps. At K = 100 and B = 20 that is 3.0 B-units at s = 20 against 50.5 at s = 1, 16.8 times less, for the same 24 GB of storage at a billion parameters.
Limits: one small MLP, plain SGD, and a hardware-noise model rather than real mixed-precision kernels; the cost model counts steps only. Log entry: LOG.md, milestone 74.