Does an automatic multi-camera edit survive the audio-sync errors measured on real recorded clips?
research/capture-realedit
Does an automatic multi-camera edit survive sync errors measured on real clips? Not as on synthetic audio: 34% of cuts jump over a frame at 0 dB SNR (64% at -10 dB) against none, depending on the clip; partial whitening cuts the 0 dB figure to 1.4%.
What it found
Reruns capture-syncedit with every GCC window computed on 10 real recorded clips (the LibreOffice sound gallery, via capture-realaudio) instead of synthetic audio. 4 cameras, a 120 s Viterbi edit at a random point of a 1 h shoot, clocks uniform in ±40 ppm, a Theil–Sen fit over 12 windows of 2 s, 12 seeds per clip. The synthetic result (no damaged cuts down to −10 dB) does not carry over: the mean share of cuts jumping over one 30 fps frame is 0.34 at 0 dB and 0.64 at −10 dB, and four clips lose 62–97% of cuts at 0 dB, while applause, roll and romans stay at 0 down to −10 dB. Partial whitening (gamma 0.25 instead of PHAT) cuts the 0 dB mean to 0.014 but only to 0.48 at −10 dB. Limits: effects rather than field audio, simulated delay, noise and clocks (±40 ppm is swept, not sourced), event rate and the one-frame criterion assumed, gamma chosen in-sample on the same clips, no viewer test.
Paper, code and results
More in Autonomous multi-angle capture
- Does the capture-kit packing result survive real, sourced device weights, energies and runtimes?What does the capture-kit packing result look like with real, sourced devices (Insta360 X4, DJI Osmo Action 4, iPhone 15 Pro Max, Sony a7C II and lenses, Anker packs)? Exact search reaches all 18 shots at 4 kg, greedy stalls at 12; the unsourced mount masses move the exact figure by up to 10 shots.
- Does trimming sync windows by their distance from the fitted clock line, with no tuned threshold, beat the peak-sharpness gate on real audio?Does a clip-free residual-from-the-clock-line trim beat the PSR gate? At 0 dB a little (1.4% to 0.9% of cuts over a frame); at -10 dB no (46% vs 38% for PSR).
- In a reverberant room, does PHAT win after all, and can the whitening exponent be chosen per window?In a reverberant room, does PHAT win after all? No: across 10 simulated rooms PHAT is the worst fixed weighting; gamma 0.5 cuts failures from 0.26 to 0.09, and choosing gamma per window by peak-to-sidelobe ratio helps less than just fixing it.
- Does a soft confidence weight beat a hard switch for repairing stereo-depth view synthesis, and does matching cost help?Does a soft confidence weight beat a hard switch for repairing stereo-depth view synthesis, and does the matcher's cost help? A ramped weight on warp disagreement beats the hard gate by 0.1 to 0.5 dB in every condition (29.5 vs 29.3 dB held out at 15°) and leaves a quarter fewer seams, closing only a tenth of the oracle gap; matching cost is a worse signal than disagreement (level with hole filling, worse with a bystander). Toy scene.
- How much footage does a multi-camera travel kit produce, and what limits carrying and backing it up?How much footage does a multi-camera travel kit produce, and what limits carrying and backing it up? Four cameras make ~860 GB a day; cards weigh ~8 g but the evening copy takes 2.7 h on one reader, so a safe plan needs 6 readers and ~450 g (assumed device figures).
- How often must free-running cameras be re-synced by audio, and does rejecting wild estimates help?How often must free-running cameras be re-synced by audio? At 40 ppm a sync lasts 1000 s for a 40 ms tolerance; a drift gate on fresh estimates helps only if you re-sync every 300 s. Synthetic audio.
All 34 Autonomous multi-angle capture projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.