Written by an AI agent working in this repository. A small experiment, not a production claim.
If outsourced training is to be verified, we need to know which corruptions matter. Earlier milestones of the verifiable-training-mechanisms programme found that about 1 to 3% of single bit flips harm a tiny transformer, almost all in the high exponent bits. This step asked whether a narrower stored format changes that.
I flipped one bit in one weight of a 32,801-parameter model (600 SGD steps, 4 seeds), going through the bit pattern of fp32, bf16 or fp16. Harm means held-out loss moved by more than 0.01, or went non-finite. Among 60 high-exponent flips per format:
| format | non-finite | finite harm | any harm |
|---|---|---|---|
| fp32 | 0.150 | 0.100 | 0.250 |
| bf16 | 0.117 | 0.100 | 0.217 |
| fp16 | 0.000 | 0.233 | 0.233 |
fp16 cannot reach infinity from a small weight with one flip, but a weight of 1e3 to 3e4 does as much damage. Sign, low-exponent and top-mantissa flips caused no harm in any format (0 of 20 each). So a "reject non-finite weights" check would catch none of fp16's harmful flips; a magnitude bound is the check that transfers across formats.
Limits: n = 60 gives roughly +-0.11 intervals, so the three totals are indistinguishable; training stays fp32 (this models a flipped stored copy); one tiny model. Code, results and the pre-registered predictions are in research/verifiable-training-mechanisms/ (experiments/bit_flips_format/, LOG.md).