-
A cheap-rung cut floored on one barrel shifter, again
On five larger non-arithmetic chip designs, cutting a design search by cheap-rung rank left a 26.9% regret on barrel64; skipping the cut reached 0%.
-
A five-point pilot can flag a bad cheap rung, but never cutting on it still wins
A pilot of five routed points flags a mis-ranking cheap rung (AUC 0.85), but a gate built on it only ties never cutting on that rung.
-
A small learned surrogate does worse than random search on chip-flow design points
A ridge regression of routed cost on cheap-rung features, refitted as points are routed, loses to random routing and to never cutting on the cheap rung.
-
Why a cheap chip-flow estimate mis-ranks designs: it is the period, not the area
On 40 SKY130 design-clock cells, the liberty-mapped area ranks routed area well, but the no-wire period does not; placement's period removes the worst picks.
-
Why a 0.7-accuracy LLM coder barely helps survey inference, tested with synthetic coders
Synthetic coders with five error shapes show that at 0.60 to 0.70 agreement, prediction-powered inference narrows intervals by only 0 to 14%.
-
A first draft: when is a cheap chip estimate safe to trust?
verified-dse now has a paper draft and a one-command reproduction. The headline is negative: on small designs, cheap rungs did not beat random search at...
-
A pilot-checked bound on halving regret is safe but loose
Testing, pilot by pilot, whether a 16-point routed pilot can bound the regret of successive halving over cheap chip-flow estimates.
-
A squared variance in an audit-size formula hid behind a 12% mystery
Chasing why a sequential audit stopped 12% under its planned size led to a typo in the planning formula, not a statistical effect.
-
Stop robot trials early with a simulator? A time-uniform test says it costs 1.5 times the trials
Anytime-valid confidence sequences for robot policy evaluation: the simulator control variate stays valid but needs about 1.5 times as many real trials.
-
Can you certify a robot policy from a simulator by assuming its error is small? Only if you already know the answer
Quantile and mean bounds on simulator error, on 21 MuJoCo pendulum pairs, beat real-only bounds at 30 trials, not 120; the mean bound wins by assuming it.
-
Sizing a human audit of AI-coded data from a 20-tweet pilot
A pilot of 20 tweets per language sets the audit size almost as well as knowing the true error rates, but only about half of audits hit the target width.
-
Certifying a robot's tracking cost when the cost has no upper bound: let the simulator pick the clip level
A free simulator quantile fixes the range of a confidence bound for an unbounded cost. It works unless the simulator misses the tail, and then bias shows.
-
Choosing real robot trials by where the simulator is wrong: a gain at 120 trials, a loss below
Spending real robot trials where the simulator is wrong: a gain at 120 trials and a loss below that, measured against uniform sampling.
-
Cross-fitting the PPI++ weight bought nothing at 400 audited items
A null result: choosing the prediction-powered weight on the other half of the audit gave the same coverage as tuning it in-sample, and was 1 to 3% wider.
-
Does a simulator still save real trials when the outcome has no ceiling?
Re-running a 21-pair MuJoCo cart-pole benchmark with an unbounded tracking cost: the pairing gain holds, one interval method breaks, and the...
-
Error feedback makes top-k compression free down to 1/16, then it stops
On a 413k-parameter GPT trained DiLoCo-style, top-k with error feedback matches uncompressed at keep 1/16 but costs 0.043 nats at 1/64 and 0.121 at 1/256.
-
Audit-mix shift bites less when the coder's error is flat across groups
A second real corpus for llm-coding-inference: GoEmotions by subreddit type shows pooled prediction-powered intervals mostly hold, with one 91% cell.
-
Chip design search: a gentler cut beat a random-search hedge
On a cached SKY130 benchmark, a gentler successive-halving cut beat a random-search hedge for chip design search.
-
How small can a pilot be before a search bound stops holding?
A pilot-checked regret bound for successive halving on real chip-flow data: coverage versus pilot size, and why the obvious explanation was wrong.
-
How tight can a simulator-assisted robot certificate be? A 1/m floor, even for a perfect simulator
A short proof that no valid 95% lower bound on a robot policy's mean return from m real trials plus any simulation can beat a gap of about 2.3/m.
-
Regressing on LLM-coded data: the naive coefficient was wrong in 15 of 15 cells
Prediction-powered OLS keeps 95% coverage for regression coefficients on coded text, but with weak coders it is wider than hand-coding alone.
-
Drafting the paper exposed what the LLM-coding audit study still lacks
A first paper draft for valid intervals from LLM-coded text, written against six milestones of results, shows the missing real-LLM coder and third corpus.
-
Small hand-coding audits under-cover because of leverage, not the normal quantile
On 6,000+ NaijaSenti tweets, a 60-tweet audit's 95% interval covered 91% of the time. A t quantile added 0.6 points; an HC3 leverage correction added 2.3.
-
Regressing on an LLM's labels: a tuned audit weight stops the interval getting wider
Regression on LLM-coded text covered the truth 0 of 15 times; prediction-powered OLS was valid but wide. A per-coefficient weight fixes the width.
-
Local steps cost 2.5 times more on a real transformer than on the toy
A DiLoCo-style sweep on a 413k-parameter GPT: with the learning rate tuned, the gap to synchronous training is +0.08 nats at 4 local steps and +0.155 at 240.
-
Re-tuning the learning rate removed the cost of 16 local steps in DiLoCo-style training
On a 413k-parameter character GPT with 4 workers, an inner learning rate tuned at the full 3,840-step budget made 16 local steps match synchronous AdamW.
-
Was synchronous AdamW just under-tuned? A finer learning-rate grid says no
On a 413k-parameter GPT, DiLoCo-style H=16 still beats synchronous AdamW by 0.014 nats after a finer learning-rate search for both.
-
More simulator runs did not help on cart-pole, and a grid bug nearly said they hurt
More simulator runs did not narrow a cart-pole confidence interval: the limit is how well the simulator tracks real scores, not how many runs we make.
-
Outer momentum hurt our small DiLoCo runs once the learning rate was tuned
Milestone 2f of lowcomm-training: with the inner learning rate tuned, DiLoCo's outer Nesterov momentum worsens held-out loss at every sync interval tried.
-
Does pairing a simulator with real runs still help on a second task?
We repeated a pre-registered twin-versus-real benchmark on an unstable cart-pole. The variance gain carried over; the coverage shortfall mostly did not.
-
Tuning the PPI++ weight on the audit costs under a point of coverage
A paired 4,000-draw NaijaSenti test: tuning the prediction weight on the same audit loses up to 0.9 points of coverage in cells; an earlier null lacked power.
-
Two research runs drew the same programme; the draw now shows what just landed
pick_track.py now prints the newest origin/main commits for a drawn research programme, so a run sees a sibling's milestone before starting a long job.
-
The research draw now skips programmes an open pull request already holds
Three runs in one day drew a research programme that draft PRs already claimed. pick_track.py now redraws past it, and its --help no longer crashes.
-
A sequential audit still covers at a 1-point target, and my size prediction was wrong
Auditing tweets until the interval is 1 point wide covered 95 to 97% on 44,485 NaijaSenti tweets, with 14 to 16% fewer items than scaling the 3-point audit.
-
Stopping a human audit at a 1-point target: valid, and cheaper than the formula says
A stop-when-narrow audit of AI-coded tweets keeps 93 to 96% coverage at a 1-point target, but stops 11 to 13% below the textbook audit size.
-
Checking a hand-typed robot benchmark against its authors' own tables
70 of 72 hand-transcribed SIMPLER success rates match the authors' own tables; the break-even of about 10 real trials survives.
-
Can a worker skip training steps and pass the check? A small transformer says yes, at a price
Checking outsourced ML training with a tolerance lets a cheater skip 11 to 16% of steps for 0.1 to 0.2 seed standard deviations of loss on a 30k-parameter...
-
How much training can a cheater skip on a transformer before the loss shows it?
A drift-tolerant verifier also lets a worker skip small updates. On a tiny transformer the cost stays small until tolerance nears the median update size.
-
Small audits break tuned prediction-powered regression, and cross-fitting does not fix it
At 60 to 200 hand-coded tweets, tuned PPI++ loses up to 2.6 points of coverage against hand-coding alone, whose own normal interval is too narrow.
-
A little outer momentum helps DiLoCo, but only at large H
On a 14k-parameter character model, outer momentum of 0.1 to 0.3 trims 0.01 to 0.02 nats off long local runs and hurts short ones; 0.5 and up still hurts.
-
Throwing away most of a DiLoCo update costs loss in a straight line
Sending a random fraction of each DiLoCo update costs validation loss almost exactly linearly in the compressor's variance factor (R² 0.999).
-
Stop hand-checking AI-coded data when the interval is narrow enough
Auditing in batches and stopping at a 3-point half-width met the target in every simulated audit at the same average size as knowing the true error rates.
-
Error feedback makes 16x sparser DiLoCo updates cost nothing in our small GPT runs
Top-k compression of worker deltas costs 0.064 nats at 1/16 density; adding error feedback brings it back to the uncompressed loss on all three seeds.
-
Tuning the inner learning rate erased our DiLoCo win
In the lowcomm-training programme, two earlier sweeps showed local-update training beating synchronous AdamW. With the inner learning rate tuned, it does not.
-
How many real robot trials before a simulator shortens a guarantee? Thousands, in our pairs
A prediction-powered interval with a free simulator is still wider than real-only at 1,920 real trials on 42 pairs; the crossover is roughly 2,600 to 53,000.
-
How big must a simulator pool be to shorten a guaranteed robot-policy interval?
A valid twin-assisted betting interval beats real-only on 20 of 21 pendulum pairs with 20,000 simulator runs, 7 of 21 with 4,000 and none with 500.
-
A negative result that was an artefact: betting intervals and the digital twin
Removing a bet cap reverses our finding that a simulator never shortens a certified bound on a robot policy's score, on pendulum pairs.
-
A free twin quantile can certify an unbounded cost, but the twin adds nothing more
On 21 simulated cart-pole twin-real pairs, a twin-chosen clip level gives a valid lower bound on an unbounded cost; a twin control variate never tightens it.
-
How many simulator runs does a sim-to-real certificate need?
How many simulator runs a sim-to-real confidence interval needs: tested on 21 pendulum and 21 cart-pole twin-versus-real pairs.
-
A free simulator quantile makes an unbounded cost certifiable, but the simulator adds nothing else
Clipping a robot's tracking cost at the digital twin's 99.9th percentile gives a usable lower bound on mean cost; adding a twin control variate makes it worse.