Can the holes and correlated errors of measured stereo depth be repaired so the in-between view beats cutting to the nearest camera?

research/capture-hybrid

Can the holes and correlated errors of measured stereo depth be repaired so in-between synthesis beats cutting to the nearest camera? A hole filler and a no-ground-truth disagreement gate that falls back to the raw nearest camera lift a 15° gap from 22.3 dB to 29.2 dB against 24.8 dB for cutting (held-out seeds 29.2); with a bystander the gate loses its edge (26.1 vs 26.3 for plain filling). Toy scene.

6 tests

What it found

capture-stereodepth found that synthesis from block-matching stereo depth never beat cutting to the nearest camera (22.3 against 24.8 dB at a 15° gap). Here the same holey, correlated depth is repaired two ways: filling holes with the farther of the nearest valid row neighbours (27.4 dB), and a gate that needs no ground truth, where the two warped sources disagree or a hole remains, falls back to the raw nearest camera (29.2 dB, 4.4 dB above cutting; the same on three held-out noise seeds). Gains hold from 8° to 30° gaps and at 6–50 cm baselines, shrink to nothing at 6% sensor noise, and with a bystander the gate loses to plain hole filling (26.1 against 26.3 dB). A per-pixel oracle at 32.0 dB bounds what any confidence rule could reach. Stylised toy scene; read the ordering, not the decibels.

A figure from its own output

Figure produced by research/capture-hybrid: gap — capture-hybrid/experiments/gap.svg, written by the project's own experiment code.

Paper, code and results

More in Autonomous multi-angle capture

Which platform can follow a travel creator through a crowd and keep the shot?Which platform can follow a creator through a crowd and keep the shot? A drone that ignores pedestrians frames 99.9% of the time but falls to 39% at 0.3 people/m² once it avoids them; a wheeled robot is stopped by step lines 42–47% of the walk, and a flat-floor one that ignores people passes within 0.5 m ~500 times an hour. 2-D simulation, assumed platform figures.
How much of a multi-camera shoot can an automatic edit recover, and what stops it?How much of a multi-camera shoot can an automatic edit recover? At detector noise 0.3 a fixed 3 s rotation shows mean true quality 0.546, a cut-cost (Viterbi) edit 0.802 vs 0.815 on true scores, and best-score-every-frame 0.723 at 345 cuts a minute; clock sync fails first on short clips (5 s at noise 0.25: 25% over 40 ms). Synthetic quality model, no real footage.
When does automatic blurring miss a bystander's face, and which filming platform is conspicuous?When does automatic blurring miss a bystander's face? On an assumed size-recall model a face detector leaves 5% of identifiable faces unblurred per frame, blurring the head of any detected body under 0.1%, and independent per-frame misses show a face in some frame of a 5 s shot 100% of the time; an offline whole-track blur leaves 0 to 5%. No real footage or detector.
Does recording only when something happens save a travel creator's battery, and what does it cost in missed moments?Does recording only on a trigger save battery, and what does it miss? With 6 thirty-second moments an hour, standby with a 10 s pre-roll uses 19.4 Wh of an 80 Wh continuous day (76% less) and captures 90% of interesting seconds; deep sleep uses 7.7 Wh but captures 69%. Misses and wake latency, not the battery, set coverage. Assumed device figures, synthetic day.
When a camera rig records only on a trigger, should every camera have its own, or should one wake the rest?When a camera rig records only on a trigger, should every camera have its own, or should one wake the rest? Four standby cameras with their own triggers use 80 Wh of 320 Wh continuous but get all four angles on only 68% of interesting seconds; one sentinel waking the others costs the same and gives 90.5%, and any-camera wake gives 100% for 92 Wh but its false-alarm energy grows with detector count. Assumed device figures.
Where in a camera ring should the sentinel go when the subject faces a favourite direction?Where in a camera ring should the sentinel go when the subject faces a favourite direction? A sentinel facing the favourite direction misses 8.6% of moments against 34.4% for the one behind; spread cameras' misses are anti-correlated, so capture-wake's independence assumption overstates the all-miss rate by 2x on a ring of 4 and understates it 12x for a tight cluster. With the turned-face miss measured in capture-detectors (54 faces) in place of the assumed curve, placement matters only if the back of a head goes undetected (untested), and weak detectors miss 8 to 45% of moments even with four cameras listening.

All 34 Autonomous multi-angle capture projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.