Does telling a learned view-synthesis blend which source is wrong help, or does it need to choose between the sources?

What it found

Follow-up to capture-learngate. Two ways to close the gap to the best per-pixel blend were tried on the same toy scenes: four new features meant to say which warped source is wrong (signed depth order, and a cross-check that warps each source into the other's camera), and a second network that predicts a per-pixel mix of the two sources before the raw-camera blend. On 40 held-out scenes (new ground textures) the new features move the learned blend from 26.78 to 26.81 dB (+0.03, ahead in 35 of 40); the source weight moves it to 27.28 dB (+0.50 over learngate, ahead in 40 of 40), recovering 0.54 of the 2.71 dB hard-gate-to-best-blend headroom against 0.36 before. The gain comes from the second output, not the new features: with the old 12 features the source weight gets 27.24 dB. Outside the training ranges the gain shrinks to +0.04 dB over learngate. Toy analytic scene, block matcher, PSNR only; every scene parameter is drawn from stated ranges that are assumed, not sourced.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Autonomous multi-angle capture

• How long a lens can a 360 camera fake by cropping into a reel, and what perspective can no crop fake?How long a lens can a 360 camera fake by cropping into a reel? An 8K equirect is native only to ~24 mm-equivalent at 1080x1920 (85 mm is a 3.5x upscale), and perspective is set by distance: a stick at 0.8 m enlarges a 10 cm-nearer feature 14% vs 7.6% at an 85 mm framing distance, so portrait looks need a camera standing back. Pixel-count upper bound. First project of track 13.
• What should a travel creator pack to film from many angles under a weight budget and the 100 Wh battery limit?What should a travel creator pack to film from many angles within a weight budget and the 100 Wh battery limit? Exact search over 17,280 kits reaches all 18 band x viewpoint shots at 5 kg; a greedy packer stalls at 12 because lenses need their body (assumed device figures).
• How many cameras, and where, to film a person from every angle?How many cameras, and where, to film a person from every angle? A 30° gap between neighbours needs 13 cameras on a ring (15° needs 25), but depth error is set by convergence angle, not count: a pair's error falls 11× from 5° to 90° (7.3 to 0.64 mm) while 3 vs 16 ring cameras only halves it. Idealised 2-D lower bounds.
• How far apart can cameras be before a reel editor can no longer synthesise the angle between them?How far apart can cameras be before a reel editor can no longer synthesise the angle between them? With exact depth a two-view warp beats cutting to the nearest camera only up to about 30° (24.8 vs 23.6 dB); beyond that holes dominate, and a bystander pulls the limit in to about 15° (ray-traced toy scene; ordering, not decibels).
• Is the depth a camera rig can actually measure good enough for the in-between angle?Is the depth a camera rig can actually measure good enough for the in-between angle? Block-matching stereo on rendered pairs feeds the two-view warp: mean depth error 14.9% at 3 cm baseline to 2.7% at 50 cm, coverage peaks at 91%, and the result never beats cutting to the nearest camera (22.3 vs 24.8 dB at a 15° gap; exact depth 27.3 dB).
• Can the holes and correlated errors of measured stereo depth be repaired so the in-between view beats cutting to the nearest camera?Can the holes and correlated errors of measured stereo depth be repaired so in-between synthesis beats cutting to the nearest camera? A hole filler and a no-ground-truth disagreement gate that falls back to the raw nearest camera lift a 15° gap from 22.3 dB to 29.2 dB against 24.8 dB for cutting (held-out seeds 29.2); with a bystander the gate loses its edge (26.1 vs 26.3 for plain filling). Toy scene.

All 34 Autonomous multi-angle capture projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.