Does a residual audio-sync error show at the cuts of an automatic multi-camera edit?

What it found

Edit quality cannot show it (a camera is shifted in score and picture together), so this counts continuity at each cut: the jump in world time between the two cameras, and loud events inside it. 4 cameras, a 120 s Viterbi edit from capture-edit at a random point of a 1 h shoot, camera clocks uniform in ±40 ppm, sync errors from the real capture-audiosync GCC-PHAT estimator on synthetic audio (12 seeds). A sync made once and never redone leaves 50% of cuts jumping over one 30 fps frame (mean jump 40 ms); a Theil–Sen fit over 12 windows leaves none from 0 dB down to −10 dB SNR with 2 s windows (−15 dB with 5 s). Below that it flips to 69–98% of cuts jumping 0.3–0.5 s with 11–24% of cuts repeating or dropping a loud event. A threshold, not a slope. Synthetic audio, assumed event rate 0.5/s and one-frame visibility criterion; ppm range swept, not sourced; no real footage and no viewer test.

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Autonomous multi-angle capture

• Does a robust fit or a Kalman filter hold a camera's clock better between audio re-syncs than a least-squares line, once the clock drift comes from a real crystal curve?Does a robust fit or a Kalman filter hold a camera clock better between audio re-syncs? On a clock rate computed from a sourced AT-cut crystal curve the plain offset hold is already fine up to 1800 s and every fit ties; only a Kalman filter tuned to an assumed 30 ppm wobble wins. Synthetic audio.
• Does finding the start offset first let short audio windows sync two drifting cameras at lower SNR?Does a coarse offset first let short windows sync drifting cameras? A 10 s window finds the offset, then 0.5 s windows hold within a frame from -12 dB SNR; plain 1 s windows fail to -5 dB. Synthetic audio.
• Does dropping doubtful sync windows, with settings chosen on other clips, keep a multi-camera edit in sync on real audio?Does a peak-sharpness gate plus settings chosen on other clips keep the edit in sync? At 0 dB yes (1.9% of cuts over a frame held-out vs 34% for PHAT); at -10 dB only partly (39% vs 64%).
• Does holding a fitted clock slope between audio re-syncs let free-running cameras go longer without a refresh?Does holding a fitted clock slope between audio re-syncs help? At 40 ppm it cuts out-of-tolerance time from 44% to 3.7% at 1800 s intervals, but an assumed rate wobble makes a strict line gate lock out. Synthetic audio; also corrects capture-syncbudget.
• Can a delay prior taken from a camera's own earlier windows give the oracle-prior gain in GCC audio sync?Can a delay prior from a camera's own earlier windows replace the oracle? Not yet: 22.5% to 37% wrong on the failing clips against 0% for the oracle, and an early alias becomes an hour-long lock-in.
• Does telling a learned view-synthesis blend which source is wrong help, or does it need to choose between the sources?Does telling the blend model which of two sources is wrong help, or does it need to choose between them? New occlusion and cross-check features add 0.03 dB; a per-pixel source weight adds 0.47 dB (40 of 40 test configs), recovering 0.54 of the headroom to the best blend.

All 34 Autonomous multi-angle capture projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.