Does marginalising old keyframes instead of pinning them fix long-path drift in a visual-inertial smoother, and does IMU grade then matter?

What it found

sensing-vio found that on 12 s paths only camera resolution moved drift, and that its smoother diverged on longer paths because the oldest pose was pinned and old velocity and bias information was forgotten. This project replaces the pin with a Schur-complement prior: after each solve, the oldest keyframe is marginalised out of its prior and the IMU factor to the next keyframe, giving a dense 8-state prior on the next one (checked against a brute-force profile in a test; the dropped keyframe's bearing factors are not marginalised). Same world, sensors and factors, 8 paired seeds, 30 s paths (about 30 m). Pinned prior: rms drift 12.3 ± 5.3 % of path at baseline IMU, 23.6 ± 7.4 % at 0.25× IMU noise, 12.3 ± 5.5 % at 4×, with worst seeds 46 to 63 %. Marginalised: 2.5 ± 0.2 % at baseline (paired ratio 0.32), 2.1 ± 1.2 % at 0.25× (median 0.8 %, one seed at 10 %), 6.5 ± 0.9 % at 4×. So with marginalisation a better IMU helps and a worse one hurts, which the pinned smoother hid. Limits: planar, point landmarks, known association, illustrative noise levels, 8 seeds, solver capped at 15 iterations, one path length, no real dataset. Noise densities are the sensing-vio values (of the order of EuRoC's published figures), not fitted.

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Sensing, imaging and 3D

• How many cheap gyros, fused, match one better gyro at the same cost?How many cheap gyros, fused, match one better gyro at equal cost? With independent errors the price ratio must beat k² (100 for a 10× quieter unit); shared error above 1/k² means no array size ties; unequal white/bias improvement makes the answer depend on integration time. Stylised.
• Does a shared temperature coefficient put a floor under an array of cheap gyros?Does a shared temperature coefficient put a floor under an array of cheap gyros? The shared part of the tempco does not average out: in the illustrative case a 3× better gyro can no longer be matched at any array size above a 0.2 °C/min ramp over 100 s; compensation quality, not array size, decides.
• How many frames from a small sensor, merged, match one frame from a bigger one?How many frames from a small sensor, merged, match one frame from a sensor with A× the pixel area? A shot-noise-limited burst needs A frames, a read-noise-limited one up to A² (formula and simulation agree); splitting the same total exposure never ties; registration blur and bracketed HDR are measured. Synthetic scenes only.
• What does a moving subject do to a burst merge, and what does robust merging cost?What does a moving subject do to a burst merge? A plain mean ghosts at every speed and gets worse as frames are added (30.5 dB at 8 frames, 25.0 at 48), median and sigma-clip do no better for a slow subject, and a reference-guided merge keeps improving and ties a 4×-area frame at about 6 frames at 0.1 dB cost when nothing moves. Synthetic scenes only.
• Can aligning a moving subject, instead of rejecting it, rescue a burst merge?Can aligning a moving subject rescue a burst merge instead of rejecting it? Inside the subject, yes (37.0 dB against 31.5 for reject-only and 32.8 for a 4×-area frame at 8 frames), but over the whole image rejecting is as good or better (31.7 against 31.8 dB), the aligner moves 10–31% of static pixels on noise, and the revealed-background ring stays 4 dB short of the large frame. Synthetic scenes only.
• Can a burst of hand-shaken low-resolution frames recover detail that merging and upsampling cannot?Can a burst of hand-shaken low-resolution frames recover detail that merging and upsampling cannot? At 2× in bright light a least-squares reconstruction reaches 29.5 dB at 16 frames against 24.0 for merge-and-upsample and 26.6 for merge-and-deblur, so 2.9 dB is sub-pixel information; it nearly vanishes in dim light and falls to nothing at about 0.27 px of shift error. Synthetic scenes only.

All 31 Sensing, imaging and 3D projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.