Can a small model trained on many random scenes choose the per-pixel blend weight for stereo-depth view synthesis better than a hand-set ramp?

What it found

A one-hidden-layer network that predicts, per pixel, how much of the raw nearest camera to blend into a stereo-depth synthesis scores 26.78 dB on 40 held-out scene configurations (new ground textures) against 26.21 dB for the best soft disagreement gate and 25.81 dB for the hard gate, ahead of the soft gate in all 40 (+0.57 dB, standard error 0.05). That recovers 0.36 of the 2.7 dB between the hard gate and the best possible per-pixel blend (soft: 0.15; ridge regression: 0.29). Outside the training ranges (gaps of 36–46°, three bystanders, heavy noise) the gain over the soft gate falls to +0.24 dB, and the matching cost adds almost nothing as a feature. Toy analytic scene, block matcher, PSNR only; every scene parameter is drawn from stated ranges, which are assumed, not sourced.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Autonomous multi-angle capture

• Does the audio-sync threshold found on synthetic noise hold on real recorded sound?Does the synthetic audio-sync threshold hold on real sound? No: on 10 real clips 2 s windows fail 53% of the time at -10 dB SNR (0% synthetic), and partial whitening instead of PHAT halves that.
• Does an automatic multi-camera edit survive the audio-sync errors measured on real recorded clips?Does an automatic multi-camera edit survive sync errors measured on real clips? Not as on synthetic audio: 34% of cuts jump over a frame at 0 dB SNR (64% at -10 dB) against none, depending on the clip; partial whitening cuts the 0 dB figure to 1.4%.
• Does the capture-kit packing result survive real, sourced device weights, energies and runtimes?What does the capture-kit packing result look like with real, sourced devices (Insta360 X4, DJI Osmo Action 4, iPhone 15 Pro Max, Sony a7C II and lenses, Anker packs)? Exact search reaches all 18 shots at 4 kg, greedy stalls at 12; the unsourced mount masses move the exact figure by up to 10 shots.
• Does trimming sync windows by their distance from the fitted clock line, with no tuned threshold, beat the peak-sharpness gate on real audio?Does a clip-free residual-from-the-clock-line trim beat the PSR gate? At 0 dB a little (1.4% to 0.9% of cuts over a frame); at -10 dB no (46% vs 38% for PSR).
• In a reverberant room, does PHAT win after all, and can the whitening exponent be chosen per window?In a reverberant room, does PHAT win after all? No: across 10 simulated rooms PHAT is the worst fixed weighting; gamma 0.5 cuts failures from 0.26 to 0.09, and choosing gamma per window by peak-to-sidelobe ratio helps less than just fixing it.
• Does a soft confidence weight beat a hard switch for repairing stereo-depth view synthesis, and does matching cost help?Does a soft confidence weight beat a hard switch for repairing stereo-depth view synthesis, and does the matcher's cost help? A ramped weight on warp disagreement beats the hard gate by 0.1 to 0.5 dB in every condition (29.5 vs 29.3 dB held out at 15°) and leaves a quarter fewer seams, closing only a tenth of the oracle gap; matching cost is a worse signal than disagreement (level with hole filling, worse with a bystander). Toy scene.

All 34 Autonomous multi-angle capture projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.