An open research programme (track 13, capture) on syncing free-running cameras and phones from their own sound: how often a short window fixes the offset to within one 30 fps frame on real devices, rooms and clocks, and whether a confidence calibrated in one room or rig keeps its failure rate in another.
Step 1 (2026-11-13) runs on LibriWASN (Schmalenstroeer, Gburrek and Haeb-Umbach 2023; Zenodo, CC BY 4.0): five smartphones (Nexus, Pixel 6a, two Pixel 6, Pixel 7, Xiaomi) recording the same loudspeaker playback of LibriSpeech, 10 ten-minute sessions in a room with about 200 ms and 10 in one with about 800 ms of reverberation. Only the phone channels are fetched, by HTTP range requests, and none is committed. For each of the 200 phone pairs a reference line (20 s GCC windows every 10 s, Theil-Sen) is checked without ground truth: every pair kept at least 97% of its windows within 5 ms of the line, and the three lines of every phone triple add up to within 1.8 ms (median 0.3 to 0.5 ms), far under the 33 ms frame.
The phones' sampling clocks differ by up to 19.1 ppm (median 13.7 ppm), the same within about 1 ppm from session to session and room to room. Short windows (0.5 to 4 s, 40 seeded times per pair and session, a simulated start-offset prior of ±1 s because the published files are pre-aligned) were scored for five weightings, with the recordings as published and with independent white noise added at 0 and −10 dB. With added noise, PHAT is the worst weighting in both real rooms and partial whitening (gamma 0.25 to 0) the best: at −10 dB, 2 s windows fail 16.9% and 33.6% of the time with PHAT against 5.0% and 7.2% with gamma 0.25 (95% session-bootstrap intervals separate), which reverses capture-reverbsync's finding in simulated rooms.
Show the rest of the summary (32 more paragraphs, 2,396 words)
The 800 ms room is harder than the 200 ms one at −10 dB, as predicted. As recorded, PHAT still fails 3.3% of 2 s windows in the 200 ms room (spread over all ten pairs at scattered wrong lags, cause open; gamma 0.25 removes nearly all of it) against a predicted under 1%. The peak-to-sidelobe ratio separates good from bad windows (AUROC 0.91 to 0.998 at 2 s), but a threshold set in one room to let at most 1% of accepted windows be wrong, applied in the other, holds on average over noise levels (0.6% to 1.6%) and not at −10 dB from the drier to the wetter room (3.7% for gamma 0.25, 5.5% for PHAT).
Limits: one dataset, one loudspeaker playback of read speech, white rather than real noise, simulated start offsets, phones not cameras; the reference's own error (closure up to 1.8 ms) bounds sub-ms claims.
Step 2a (2026-11-13) tests the threshold theory of time-delay estimation (H4). Ianniello's (1982) gross-error model is extended to weighted GCC (weight (GxxGyy)−γ/2), fed with each window's Welch spectra, and checked on 84,000 simulated windows: Gaussian signals, and real read speech (MS-SNSD, MIT) with white or real babble noise in dry rooms and statistical rooms of RT60 0.2 and 0.8 s. zenodo.org was unreachable from that session, so the LibriWASN check is step 2b, with its harness committed.
On speech, the model predicts the share of windows more than a frame off within a factor of 2 in 137 of 171 scored cells (80%, against a pre-registered 75%). It runs low (median predicted/observed 0.58 to 0.74), misses most at 0 dB with 0.5 s windows, and agrees in 82 of 83 cells at −15 dB. The Gaussian check agreed in 32 of 36 cells, narrowly missing its pre-registered 90%.
Step 1's GCC normalises each FFT bin by its own magnitude, and that costs PHAT: averaged over the speech cells it misses 39.7% against 32.6% for the same weighting from smoothed spectra. With real babble the weighting ranking flips: PHAT is best and plain correlation worst (0.8 s room, −10 dB, 2 s: 50% against 79%).
Step 1's preference for partial whitening therefore holds for white noise only. As a blind confidence, the model's probability flags PHAT failures better than the peak-to-sidelobe ratio (median AUROC 0.85 against 0.74), but not for gamma 0.25.
Limits: the rooms are statistical, the delays are simulated, the babble is independent per device rather than shared, and the H4 score uses spectra aligned at the true lag.
Step 2b (2026-11-14) runs the same model window for window on the LibriWASN phones of step 1 (96,000 windows; the replay reproduced all 96 of step 1's miss rates exactly). H4 is refuted on real devices: the model is within a factor of 2 in 41 of 94 scored cells (44%, against the pre-registered 75%; Rice's variant 39/94). It now runs high (median predicted/observed 2.2, no cell underpredicted by more than x2), and its error grows with window length: 24 of 24 cells agree at 0.5 s, 1 of 24 at 2 s and 2 of 22 at 4 s, for every weighting.
The 800 ms room agrees more often than the 200 ms one (52% against 35%), and raw-bin PHAT again misses more than smooth-weight PHAT at −10 dB (all 8 window-room cells). Why real recordings improve with window length faster than the theory says is open.
Step 3 (2026-11-14) puts a distribution-free threshold on the peak-to-sidelobe ratio: Learn then Test (Angelopoulos et al. 2021), set for at most 1% wrong among accepted windows at delta 0.1, calibrated in one condition and tested in another on the same 96,000 windows. For 2 s windows with gamma 0.25, room to room gives 1.45% and 0.57% wrong among accepted (keeping 97 to 98%), and leaving each phone out gives 0.49% to 1.30%; but calibrating on the recordings as published and at 0 dB and testing at −10 dB gives 4.65% and 7.25%, because on cleaner audio the threshold falls to the floor.
A gate calibrated on a mix of noise levels controls the mix, not the noisiest part. Requiring the PHAT and gamma 0.25 estimates to agree (post hoc) cuts the noise-shift failure to 1.43% and 2.57% at 2 s, though not at 1 s. Within one room 13% of session splits exceed 1% (the i.i.d. bound is 10%), because windows cluster by session. Two rooms, five phones and added white noise only.
Step 3b (2026-11-14) tests two noise-aware gates on the same windows. A blind SNR estimate, the spread of a window’s 32 ms frame energies, separates the noise levels almost completely (2 s medians 21, 5.2 and 1.2 dB as recorded, at 0 dB and at −10 dB). Gating with one Learn-then-Test threshold per calibration quartile of that estimate, and refusing windows below the calibration’s range, keeps wrong-among-accepted under 1% in every shift: 0.27% room to room at −10 dB (keeping 74%, against 3.38% for the pooled gate), and 4% of within-room session splits above 1% (pooled 15%).
Under the unseen noise level it lets nothing wrong through, by refusing 93% and 99% of windows, most of them right. Scoring windows with the threshold theory instead of PSR is a slightly better flag for PHAT (AUROC 0.987 against 0.977) but does not carry the bound (2.25% and 5.37% under the noise shift). The recipe: cover the noise range in calibration, gate per blind-SNR bin, refuse below it. Added white noise only.
Step 3c (2026-11-15) asks whether that recipe survives real noise. With LibriWASN out of reach, it uses real read speech (MS-SNSD, MIT) in statistical rooms (RT60 0.2 and 0.8 s), 186,000 windows, each clean pair scored under white noise and nine kinds of real recorded noise (air conditioner, airport announcements, babble, copy machine, munching, neighbour, door, typing, vacuum cleaner) at +10, 0 and −10 dB, with each device's noise drawn from files the other never hears. It does not survive.
The blind SNR estimate reads real noise at −10 dB as 3.3 to 16.6 dB (white: 1.1 dB), because frame-energy spread measures how steady the noise is, not how loud. So the gate calibrated on white noise accepts 99–100% of 2 s windows and lets 2.6% to 81% wrong through, for every real type in both rooms. Calibrating on the other nine types, then testing on the held-out one, mostly refuses everything (the pooled gate keeps nothing), yet still passes 80% and 93% wrong windows when the held-out noise is typing.
Requiring PHAT and partial whitening to agree holds for steady noise (0.0–0.3% wrong for air conditioner, airport and vacuum cleaner) but not for impulsive noise (87–90% for typing). No fixed weighting wins: at −10 dB PHAT misses 2% of 2 s windows under air-conditioner noise against 12% for gamma 0.25, but 84% against 30% under copy-machine noise. Two reruns were needed before the result, both logged: the first let the two devices share noise samples, and the second missed duplicated noise files.
Step 3d adds blind features that name the noise (frame-energy kurtosis and spectral flatness) as Mondrian bins with conformal novelty refusal: on held-out noise types the gate keeps 43–48% of windows instead of 10–18%, but meets 2% at −10 dB for only 3–5 of 10 types (Neighbor 66–75% wrong, Typing 86–97%), because 92–100% of a new type's windows look like the library's. Calibrated on the site's own noise it holds the bound pooled over noise levels (9–10 of 10 types) but not at −10 dB alone.
Limits: simulated rooms and delays, and noise independent per device rather than one source heard by both.
Step 3e (2026-11-16) gave the gate the 4 s before each window: a minimum-statistics noise floor (a simplified Martin 2001 tracker) as the bin. Under site calibration at −10 dB it met the 2% bar for 4 of 10 noise types in each room, the same as the window-only estimate; for neighbour noise, door slams and typing the window’s own estimate rates −10 dB windows as less noisy than 0 dB ones (AUROC 0.36–0.47), and the context lifts that only to 0.48–0.66.
An oracle that knows the true level finds no threshold that certifies 1% for 7 of 10 types at −10 dB and refuses them, so the guarantee a single-microphone rig can state is the failure rate over its own mix of noise, with wrong windows at most 2.4% of the worst stretch here.
Step 3f (2026-11-17) lets both devices hear one noise source, placed with the talker at random around devices 3, 10 or 30 m apart, and scores errors against the clock rather than the talker. Any sound both devices hear arrives within D/c of the clock offset, so at 30 fps a lock on shared noise cannot cost a frame while the devices are under 11.4 m apart. Inside that bound shared noise is signal: at −10 dB, 2 s windows miss 0% under one shared babble source against 59–64% with independent babble, shared noise beats independent noise for 9 of 10 types in each room, and the gate keeps 60–63% of windows instead of 23–24%.
The exception is noise that repeats inside the search range: three looped door-slam recordings give equal peaks, 24–27% of windows miss, and a gate calibrated without shared noise passes 19–20% wrong. At 30 m the talker’s own path difference exceeds a frame in 69% of layouts, so about 70% of windows miss at every noise level and the gate refuses everything; a spread-out rig needs a clap equidistant from its cameras or a correction for known positions. Free-field geometry with statistical impulse responses; noise fully shared or fully independent, not mixed.
Step 3g replays those 242,000 windows bit for bit with the second-peak ratio (runner-up correlation more than a frame from the peak, over the peak) as the confidence: calibrated on independent noise and tested on shared noise at 3 m, the margin (1 − ratio) passes 1.1% and 0.0% wrong among accepted door windows where PSR passed 19% and 20%, keeps 69% and 68% of all windows against 63% and 60%, and cuts wrong-and-accepted windows from 1.4–1.5% to 0.0–0.1% of the total. It does not help impulsive noise heard by one device only (typing, munching, neighbours): the 2% bound holds for 4–5 of 10 types whichever score is used, because there the wrong peak has no close rival.
Step 3h replays step 3c’s 62,000 windows bit for bit with GCC on envelope-normalised audio (each device’s window divided by its own 32 ms moving RMS): at −10 dB with noise heard by one device only, windows more than a frame off fall from 78% to 0.1% under typing, 30% to 0.1% under munching, 21% to 0.2% under a door and 67–74% to 8–9% under neighbours, and the 1% gate on its own PSR keeps 84–91% of impulsive-noise windows (plain PSR kept 4–5%) while meeting 2% for 9 of 10 types (plain 4).
Steady noise gets worse (air conditioner 12–16% to 21–23%, vacuum 3% to 6–8%), so a per-window choice between the two estimators is the next step. A sub-window consistency gate fails (5–7 of 10 types), because periodic keystrokes align at the wrong lag throughout the window.
Step 3i fits a per-window switch between plain and normalised GCC on the window’s frame-energy kurtosis: read speech is itself bursty, so the fitted threshold sends 94–100% of windows to the normalised estimator and the switch does no better than always normalising (mean failure over ten noise types at −10 dB 11.5% and 12.5% against 11.3% and 12.5%; plain 33–35%); choosing by the higher PSR is worse (17.5–18.9%), and a per-window oracle would reach 7.8–8.9%, mostly on babble.
Step 3j (2026-11-18) checks steps 3c-3i's simulated room against the real phones: the same nine MS-SNSD noise types and white noise added independently per phone to step 1's 2 s LibriWASN windows (168,000 windows; step 1's windows replayed bit for bit, 6 of 6 cells equal). Across the ten noise types the real −10 dB failure shares rank like the simulated ones (Spearman 0.98 and 1.00 for gamma 0.25 GCC, 0.89 and 1.00 for the envelope-normalised estimator), and the sign of normalisation's effect agrees for 9 and 8 of 10 types.
Normalisation takes typing from 65.7% and 79.1% of windows a frame off to 2.8% and 1.3% and halves the failure summed over types (2.78 to 1.33, 4.01 to 2.11), at a steady-noise cost larger than simulated (air conditioner +7.4 and +10.5 points). The twin is optimistic about level: the normalised estimator fails 13% and 21% of real windows on average against 11% and 13% simulated, and white noise 5-10% against under 1%. A single 1% gate on the normalised estimator's PSR keeps 78% and 65% of real −10 dB windows and holds 2% for 7 and 6 of 10 types (misses are speech-like noise); on plain GCC it refuses nearly everything.
Limits: noise added rather than recorded in the room, one loudspeaker playback, phones not cameras.
Step 3k (2026-11-19) fits the twin to the phones on white noise only, choosing its direct-to-reverberant ratio and a white noise floor on real sessions 0-4, and tests it on the nine noise types it never saw (124,000 simulated windows, step 3h's replayed 62 of 62). No setting matches: in the twin reverberation is either harmless or breaks noise-free windows too (8–9% at DRR −6 dB in the 800 ms room against 0.1–0.4% real), so the fitted twin's mean gap over the nine types grows from 5.0 to 20.4 points in the 200 ms room and stays 7.4 in the 800 ms room, though it still ranks the types (Spearman 0.94–1.00).
A 1% gate set wholly on twin windows nonetheless holds 2% wrong among accepted on real test sessions for 7 and 6 of 10 types, keeping 77% and 68%, as well as the gate set on real recordings (7 and 6, 78% and 65%); the fitted, too-reverberant twin's gate holds for 10 of 10 types in the 200 ms room (0.20% wrong overall, keeping 67%), and no twin fixes speech-like noise in the 800 ms room. Post hoc, one real window in twenty has only 52–59% of its power in GCC's band against 89–90% simulated, a lead for a per-phone microphone model.
Limits: a coarse two-knob grid, a white rather than recorded floor, the same dataset as step 3j.
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.