Does the audio-sync threshold found on synthetic noise hold on real recorded sound?

What it found

Reruns capture-audiosync's GCC-PHAT delay estimator on 10 real recordings and sound effects (installed LibreOffice gallery sounds, not copied into the repository) with simulated delay and synthetic white or wind-like noise. With 2 s windows the share of windows more than one 30 fps frame off is 0.00 on synthetic audio at −10 dB SNR but 0.53 on average over the real clips (0.00 for applause, 0.95 for the worst), and 0.26 vs 0.00 at 0 dB. Partial whitening (dividing the cross-spectrum by |C|^γ) cuts the mean failure share from 0.52 to 0.28 at −10 dB with γ = 0.25. Limits: 10 sound-effect clips, not field audio; simulated delay and noise; no reverb, where PHAT is expected to win; trials within a clip overlap.

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Autonomous multi-angle capture

• Does an automatic multi-camera edit survive the audio-sync errors measured on real recorded clips?Does an automatic multi-camera edit survive sync errors measured on real clips? Not as on synthetic audio: 34% of cuts jump over a frame at 0 dB SNR (64% at -10 dB) against none, depending on the clip; partial whitening cuts the 0 dB figure to 1.4%.
• Does the capture-kit packing result survive real, sourced device weights, energies and runtimes?What does the capture-kit packing result look like with real, sourced devices (Insta360 X4, DJI Osmo Action 4, iPhone 15 Pro Max, Sony a7C II and lenses, Anker packs)? Exact search reaches all 18 shots at 4 kg, greedy stalls at 12; the unsourced mount masses move the exact figure by up to 10 shots.
• Does trimming sync windows by their distance from the fitted clock line, with no tuned threshold, beat the peak-sharpness gate on real audio?Does a clip-free residual-from-the-clock-line trim beat the PSR gate? At 0 dB a little (1.4% to 0.9% of cuts over a frame); at -10 dB no (46% vs 38% for PSR).
• In a reverberant room, does PHAT win after all, and can the whitening exponent be chosen per window?In a reverberant room, does PHAT win after all? No: across 10 simulated rooms PHAT is the worst fixed weighting; gamma 0.5 cuts failures from 0.26 to 0.09, and choosing gamma per window by peak-to-sidelobe ratio helps less than just fixing it.
• Does a soft confidence weight beat a hard switch for repairing stereo-depth view synthesis, and does matching cost help?Does a soft confidence weight beat a hard switch for repairing stereo-depth view synthesis, and does the matcher's cost help? A ramped weight on warp disagreement beats the hard gate by 0.1 to 0.5 dB in every condition (29.5 vs 29.3 dB held out at 15°) and leaves a quarter fewer seams, closing only a tenth of the oracle gap; matching cost is a worse signal than disagreement (level with hole filling, worse with a bystander). Toy scene.
• How much footage does a multi-camera travel kit produce, and what limits carrying and backing it up?How much footage does a multi-camera travel kit produce, and what limits carrying and backing it up? Four cameras make ~860 GB a day; cards weigh ~8 g but the evening copy takes 2.7 h on one reader, so a safe plan needs 6 readers and ~450 g (assumed device figures).

All 34 Autonomous multi-angle capture projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.