Research topic
Sensing, imaging and 3D
Getting more from cheap sensors: fusing cheap IMUs and cameras, computational imaging, and 3D models from real captures, with the error floors that correlated errors set.
31 projects, each with its own page, code and tests. See them on the full research index.
research/sensing-imuarray · 8 tests
How many cheap gyros, fused, match one better gyro at equal cost? With independent errors the price ratio must beat k² (100 for a 10× quieter unit); shared error above 1/k² means no array size ties; unequal white/bias improvement makes the answer depend on integration time. Stylised.
Abstract
Single-axis gyro model with angle random walk and rate random walk (Allan-variance terms) and a fraction ρ of each shared between units. With independent errors an array of n cheap units ties a unit k times quieter only when the price ratio exceeds k2 (9 for k = 3, 100 for k = 10); shared error above 1/k2 means no array size ties (k = 10, ρ ≥ 0.01). If the better unit improves white noise 10× but bias walk only 3×, the number of units needed falls from 100 at 1 s to 10 beyond 300 s, so the answer depends on integration time. Monte Carlo matches the closed form within 0.03. Stylised: illustrative grades, not datasheets; no real IMU data; no prices claimed.
research/sensing-imutemp · 9 tests
Does a shared temperature coefficient put a floor under an array of cheap gyros? The shared part of the tempco does not average out: in the illustrative case a 3× better gyro can no longer be matched at any array size above a 0.2 °C/min ramp over 100 s; compensation quality, not array size, decides.
Abstract
Follow-up to sensing-imuarray. Each gyro has a temperature coefficient ci = μ + s·ei; under a temperature ramp the shared mean μ leaves an angle-error floor μrT2/2 that no array size removes. In the illustrative case (100 s) a 3× better unit can be matched at 10–15 units up to 0.1 °C/min but at no size from 0.3 °C/min; a 10× better one at none from 0.1 °C/min. Compensating both grades equally brings the array size back near k2 only at about 2% residual tempco. Monte Carlo matches the closed form within 1%. Stylised: assumed tempcos, no real IMU data, no prices claimed.
research/sensing-burst · 12 tests
How many frames from a small sensor, merged, match one frame from a sensor with A× the pixel area? A shot-noise-limited burst needs A frames, a read-noise-limited one up to A² (formula and simulation agree); splitting the same total exposure never ties; registration blur and bracketed HDR are measured. Synthetic scenes only.
Abstract
Synthetic scenes with Poisson shot noise, Gaussian read noise and a full-well clip. A burst of N small-sensor frames (each at the large frame's exposure time) ties a sensor with A× the pixel area at N* = A²(sx+r²)/(Asx+r²): A frames when shot noise dominates, up to A² in the dark or at high read noise (simulation, A=4: 4.0 frames in bright light, 14.4 in dim light with 8 e− read noise). Splitting the same total exposure into frames never ties: the burst has more variance than the large frame at every N. With hand shake and Fourier-shift registration, shifts are recovered to 0.09–0.8 px RMS in good light but fail in the dimmest case tested (3.1 px), and the registration blur means A=4 needs about 21 frames in bright light. A bracketed burst recovers clipped highlights (42 dB against 22 dB for one frame). Stylised: synthetic images, global translation only, no real raw files.
research/sensing-ghost · 7 tests
What does a moving subject do to a burst merge? A plain mean ghosts at every speed and gets worse as frames are added (30.5 dB at 8 frames, 25.0 at 48), median and sigma-clip do no better for a slow subject, and a reference-guided merge keeps improving and ties a 4×-area frame at about 6 frames at 0.1 dB cost when nothing moves. Synthetic scenes only.
Abstract
Still camera, one disc moving across a synthetic burst (Poisson shot noise, Gaussian read noise, a large sensor with 4× the pixel area). A plain mean merge ghosts at every speed tested and gets worse as frames are added (30.5 dB at 8 frames, 25.0 dB at 48 for 0.25 px per frame, against 30.8 dB for one large-sensor frame); median and sigma-clip merges do no better because a slow subject covers each pixel in most frames. A reference-guided merge (add a frame only where it agrees with frame 0) keeps improving, to 37.6 dB at 48 frames, and ties the large frame at about 6 frames; with nothing moving it costs 0.1 dB against the mean, the median 1.3 dB. Stylised: synthetic images, one translating disc, a still camera, untuned thresholds, no real raw files.
research/sensing-ghostalign · 7 tests
Can aligning a moving subject rescue a burst merge instead of rejecting it? Inside the subject, yes (37.0 dB against 31.5 for reject-only and 32.8 for a 4×-area frame at 8 frames), but over the whole image rejecting is as good or better (31.7 against 31.8 dB), the aligner moves 10–31% of static pixels on noise, and the revealed-background ring stays 4 dB short of the large frame. Synthetic scenes only.
Abstract
Still camera, one disc moving across a synthetic burst (Poisson shot noise, Gaussian read noise, a large sensor with 4× the pixel area). A per-pixel sub-pixel alignment to the reference frame, followed by a rejection test, reaches 37.0 dB inside the disc at 8 frames and 1 px per frame, against 31.5 dB for reject-only merging and 32.8 dB for the large frame. Over the whole image it is no better than rejecting (31.7 against 31.8 dB at 8 frames, 34.4 against 35.6 at 24) because the aligner moves 10–31% of static pixels on noise, and in the ring where the background is covered and revealed no variant gets within 4 dB of the large frame. Stylised: synthetic scenes, one flat rigid disc, translation only.
research/sensing-superres · 10 tests
Can a burst of hand-shaken low-resolution frames recover detail that merging and upsampling cannot? At 2× in bright light a least-squares reconstruction reaches 29.5 dB at 16 frames against 24.0 for merge-and-upsample and 26.6 for merge-and-deblur, so 2.9 dB is sub-pixel information; it nearly vanishes in dim light and falls to nothing at about 0.27 px of shift error. Synthetic scenes only.
Abstract
Synthetic 32×32 frames from a box-averaging sensor (Poisson shot noise, 3 e− read noise), shifted by up to two pixels, reconstructed on a 2× grid by regularised least squares. In bright light it reaches 29.5 dB at 16 frames and 30.7 dB at 32, against 24.0 dB for register, average and bilinear upsample and 26.6 dB for register, average and deconvolve: 2.6 dB of the gain is deblurring the pixel box and 2.9 dB (4.0 at 32 frames) is sub-pixel information. In dim light that part is within ±0.3 dB; registration error of 0.27 LR pixel erases it. Against a sensor with 2× the linear resolution and the same total light the burst wins at 2 frames and loses from about 4; at 3× it ties at 16 frames and at 4× it wins. Synthetic scenes, translation only, no mosaic or lens blur; the reconstruction is plain Tikhonov, not a published method.
research/sensing-timesync · 7 tests
How many milliseconds of camera-to-gyro time offset double a fused attitude error, and can an alignment fit recover it? About 12–14 ms at 30 deg/s motion; the fit recovers it to ~1 ms with motion and fails on a still platform. Synthetic, single axis.
Abstract
Single-axis simulation: a 200 Hz biased gyro and a 30 Hz camera whose timestamps are late by td, fused by a complementary filter. At 30 deg/s motion the error doubles at 12–14 ms; the doubling offset is 32–34 ms at 10 deg/s and 8–10 ms at 90 deg/s, and grows with camera noise. A gyro-integral alignment fit recovers the offset to about 1 ms rms and restores the zero-offset error, but fails on a nearly still platform. Stylised: one axis, constant offset, angle measurements only, synthetic noise.
research/sensing-syncskew · 6 tests
How much motion does a camera-to-gyro offset fit need, and what does clock skew do to a one-off calibration? A 20 ms offset needs about 16 deg/s rms motion to fit reliably; 1000 ppm skew leaves 0.254 deg error after a fixed-offset fit against 0.155 deg when skew is fitted too. Stylised, synthetic.
Abstract
Same single-axis gyro and camera world as sensing-timesync, with the camera stamp error growing as td = td0 + skew·t. An alignment fit recovers a 20 ms offset within 5 ms in 8 of 8 runs at 16 deg/s or more, but only 2 of 8 at 1 deg/s and 0 of 8 at 0.3 deg/s. A camera clock 1000 ppm fast raises attitude error from 0.152 to 0.514 deg uncorrected and 0.254 deg after a fixed-offset fit, while fitting offset and skew together gives 0.155 deg up to 2000 ppm; skew recovery needs a window of about 40 s. Stylised: one axis, linear skew, synthetic noise, skews exaggerated to show an effect in 40 s.
research/sensing-scan2cad · 12 tests
How accurate must a scan be to rebuild a part as parametric CAD? On synthetic plate scans, RANSAC plane and circle fits give all five dimensions within 0.2 mm for noise up to 0.2 mm, but at 0.5 mm noise the error is a bias (thickness −0.20 mm) that more points do not remove, and a 0.5% scale error alone moves an 80 mm length by 0.40 mm. Synthetic, one shape.
Abstract
Synthetic point-cloud scans of an 80×40×10 mm plate with two 8 mm holes, fitted by RANSAC planes and circle fits. With scan noise of 0.2 mm or less every scan gives all five dimensions within 0.2 mm at 2 000 points or more (worst RMS 0.09 mm). At 0.5 mm noise only 0–50% of scans are usable at any point count because the error is a bias: thickness reads 0.20 mm low and the bias grows with the inlier band (−0.07 to −0.38 mm for 1.5σ to 4σ). A scale error of 0.5% from a reconstruction with no scale reference already moves an 80 mm length by 0.40 mm. Stylised: one shape, assumed Gaussian noise, pre-registered scan, generated CadQuery script not run.
research/sensing-recon · 11 tests
How many photos does a usable 3-D reconstruction need? On a synthetic object with known cameras, 12 views on a full orbit triangulate every feature, 8 views 89% and 4 or fewer at most 13%; per-point error then depends on pixel noise and pose error, not view count. Triangulation stage only (synthetic).
Abstract
Synthetic bumpy-ellipsoid object, 800 features, pinhole cameras on an orbit with known poses and linear (DLT) triangulation. On a full orbit 12 views reconstruct every feature, 8 views 89%, 6 views 59% and 4 or fewer at most 13%, because a point must be seen by two cameras. Once triangulated, error is about 0.0028 object radii per pixel of image noise, a 1° pose error does the damage of about 2 px, and a narrow baseline (5° span) raises error about 7× against a 40° span. Triangulation stage only: matches and poses are given, there is no occlusion, mesh or splat, and no real images.
research/sensing-thermalfuse · 10 tests
Does a low-resolution thermal camera find people in darkness and smoke, and what does misalignment cost? On synthetic scenes RGB finds 36% in bright light and none in the dark or dense smoke, thermal alone 65% everywhere (capped by warm decoys), fusion 94% in bright light and the thermal level otherwise; a 4 px (one thermal pixel) displacement costs the fused detector 19 points and 6 px erases the gain. Synthetic only.
Abstract
Synthetic 64×64 scenes with people, cold clutter and warm non-person decoys, seen by a grey RGB camera (shot and read noise, smoke airlight) and a 16×16 thermal camera (0.3 K noise, 10% of the visible smoke extinction assumed). At 10% false alarms per image, RGB alone finds 36% of people in bright clear light, 13% at smoke transmittance 0.4 and none in the dark or dense smoke; thermal alone finds 65% in every condition, capped by the decoys; fusing them gives 94% in bright light and falls back to the thermal level otherwise. A thermal camera displaced by 2 px costs the fused detector 7 points, 4 px (one thermal pixel) 19, and 6 px leaves it below RGB alone (0.12 vs 0.37); training with displacement jitter keeps 0.54 at 6 px but loses 9 points when aligned. Stylised: synthetic scenes, logistic-regression detectors on local contrast, sideways shift only, no real data.
research/sensing-vio · 8 tests
Which upgrade buys the most visual-inertial accuracy, a sharper camera or a better IMU? On a planar synthetic smoother only camera resolution moved drift (1280 px 1.2 % vs 1.7 % at 640 px); a 0.25–4× change in IMU noise did not, on 12 s paths. Stylised, not EuRoC.
Abstract
A planar synthetic visual-inertial smoother (bearing landmarks, 100 Hz gyro and accelerometer) on 12 s paths, 8 paired seeds. Camera resolution moved drift: 1.18 % of path rms at 1280 px against 1.65 % at 640 px and 8.9 % at 320 px. Scaling IMU noise 0.25× to 4× changed it by under 10 %, inside the noise. Limitations: the smoother has no marginalisation and diverged on 30 s paths, so the IMU result is for short runs only; planar, synthetic, not EuRoC.
research/sensing-ba · 6 tests
With 12 views, 0.5 px noise and every camera pose wrong, bundle adjustment cuts mean surface error from 0.0117 to 0.0014 object radii (the true-pose floor), 8× here and 73× at 0.2 radii of position error, but only 1.4× at 4 px of pixel noise and not at all with 4 views. Synthetic only
Abstract
Follows sensing-recon, which found a 1° camera pose error costs as much as about 2 px of pixel noise. On the same synthetic bumpy-ellipsoid rig, every camera gets a random rotation (0–4° per axis) and position (0–0.2 object radii) error; sparse bundle adjustment (SciPy least_squares) then refines all poses and points from the wrong-pose triangulation, and both are scored after a similarity alignment. With 12 views, 0.5 px noise, 0.5° and 0.02 radii the mean surface error falls from 0.0117 to 0.0014, equal to triangulation with true poses; the gain is 73× at 0.2 radii of position error but only 1.4× at 4 px noise (where pixel noise sets the error) and 0.9× with 4 views. A pose prior with the true sigma changed nothing. Limits: synthetic, matches given and correct, no outliers, known intrinsics, a start close to the truth, and alignment to the true points; 8 seeds per cell.
research/sensing-bafocal · 6 tests
With 12 views, wrong poses and a focal length told 5% wrong, a fixed-focal adjustment costs 1.2× the true-focal error (2.7× at 20%); solving for the focal returns the floor with the focal within 0.3%, and for k1 = −0.3 distortion 0.0034 falls to 0.0015, but not at 4 views or above 2 px noise. Synthetic only
Abstract
Follows sensing-ba and sensing-baoutlier, which assumed known intrinsics. On the same synthetic rig (12 views, 0.5 px noise, every pose wrong) the solver is told a focal length off by −20% to +20%. With the focal held, the adjustment's mean surface error is 0.0015 at ±2%, 0.0018 at ±5% and 0.0039–0.0049 at ±20%, against 0.0014 for the true focal; adding one shared focal length to the unknowns gives 0.0014 from every start and recovers it to 0.3%. With k1 = −0.3 radial distortion unmodelled the error is 0.0034; solving for k1 gives 0.0015. The gain is 3.6× at 0.1 px of noise but about 1× from 1 px, and with 4 views none of the variants beat 0.0068 and the focal is 2.4% off. Limits: synthetic, the fitted distortion law is the rendering law, one shared focal, no outliers; the focal and k1 ranges are swept, not measured from a device; 6 seeds per cell.
research/sensing-bak2 · 5 tests
Rendering with k1 = −0.2 and k2 swept from −0.2 to +0.4, a bundle adjustment solving for the focal and k1 only returns 0.0015 object radii, the same as one that also solves k2, against 0.0022–0.0030 with no distortion model; the lens parameters are not separately identified (k1 stays near −0.16, k2 sits about 0.16 low) and the gain vanishes by 2 px noise. Synthetic only
Abstract
Follows sensing-bafocal, whose fitted radial law was the rendering law, and takes its “model mismatch” next step. Images are rendered on the sensing-ba rig (12 views, 0.5 px noise, every pose wrong) with xd = x(1 + k1r² + k2r⁴). With k1 = −0.2 and k2 from −0.2 to +0.4, an adjustment with no distortion model costs 0.0024–0.0030 (focal free) and 0.0022–0.0027 (focal held), while fitting the focal and k1 alone gives 0.0015 in every case, equal to fitting k2 as well. The parameters are not recovered separately: the k1 estimate is −0.158 whatever the true k2 and the k2 estimate moves with the truth but sits about 0.16 below it, and the focal ends 1.1% off; surface error is insensitive to this. The benefit disappears by 2 px of noise (0.0061 for all three), and at 6 views the focal is 4% off. Limits: synthetic, the lens-parameter ranges are swept, not measured from a device; one object; correct matches; 6 seeds per cell; what the one-term model gets right is the surface, not the lens parameters.
research/sensing-bak2wide · 4 tests
Moving the camera in and shortening the focal length (800 to 400 px) raises the largest normalised radius from 0.34 to 0.85, and the spread of the fitted k2 over seeds falls from 0.33 to 0.02 with its mean within 0.01 of truth; the surface then needs k2 (0.0027 with k1 only against 0.0020 with both). Synthetic only
Abstract
Takes the next step of sensing-bak2, where k2 was invisible on a narrow field of view. The camera is moved in (distance 4 to 2) and the focal length shortened with it (800 to 400 px) so the object keeps its pixel size, which raises the largest normalised radius of a visible point from 0.34 to 0.85. The standard deviation of the fitted k2 over 6 seeds falls from 0.33 to 0.02 and its mean is within 0.01 of the true +0.1 from distance 2.5 on; k1 is recovered to −0.195 against a true −0.2. The surface now needs the second term: at distance 2 a k1-only adjustment costs 0.0027 object radii against 0.0020 with k2 (0.0055 against 0.0022 at k2 = −0.2), and ignoring distortion costs 0.0082–0.0118. At 2 px of noise the two fits meet at 0.0081–0.0083. Limits: synthetic, lens values swept not measured, one object, the closer camera adds perspective so field of view and range are not separated, no decentring or principal-point error, correct matches, 6 seeds.
research/sensing-baoutlier · 5 tests
With 12 views and wrong camera poses, 2% of observations replaced by random pixels already raise bundle-adjustment surface error from 0.0014 to 0.079 object radii, and Huber or Cauchy loss from the usual all-view start only halve that; a per-point RANSAC start fixes 5% (0.0319) but only 24 views get back to the clean floor at 10% (0.0012). Synthetic only
Abstract
Follows sensing-ba, whose next step was matching with outliers and a robust loss. On the same synthetic rig (12 views, 0.5 px noise, every pose wrong) a fraction of the observations is replaced by a uniform random pixel. Plain bundle adjustment started from all-view DLT triangulation degrades from 0.0014 (clean) to 0.079 object radii at 2% mismatches and 0.52 at 10%; Huber and Cauchy loss (SciPy least_squares) help only partly (0.027 and 0.043 at 2%, 0.37 and 0.38 at 10%), and trim-and-refit does nothing, because the contaminated all-view triangulation is already a bad start. A per-point RANSAC pair triangulation (20 px gate, wide because the poses are wrong) as the start brings 5% mismatches to 0.032 and, with Cauchy loss, only with 24 views reaches the clean floor at 10% (0.0012 against 0.0010); with 8 to 12 views, where a point is seen in about three, it cannot vote and fails. A robust loss with 40 function evaluations stalled before converging (300 were used). Limits: synthetic, uniform random-pixel mismatches only, known intrinsics, 6 seeds per cell, one gate and loss scale each, not tuned.
research/sensing-bapp · 3 tests
A 10 px principal-point offset held at the image centre raises the surface error from 0.0019 to 0.0043 object radii on the wide rig (focal 3.5% off) against 0.0019 to 0.0015 fitted on the narrow one; fitting the point recovers it and removes the cost. Synthetic only
Abstract
Takes the next step of sensing-bak2wide. Images are rendered with a true principal-point offset of 0–20 px; one adjustment holds the point at the image centre, the other fits it, both with focal length, k1, k2, poses and points free. On the wide rig (distance 2, f = 400 px) a 10 px offset raises the surface error from 0.0019 to 0.0043 object radii when held (0.0079 at 20 px) and leaves the focal length 3.5% off (10% at 20 px); fitting the point holds 0.0019 throughout and recovers dx to within 0.1 px. The narrow rig is far less sensitive (0.0019 against 0.0015 at 10 px), so a wider view makes the offset matter more. At 2 px of pixel noise the benefit of fitting disappears (0.0083 against 0.0079). dy is pulled toward zero (−7.1 for a true −8). Limits: synthetic, offsets swept not measured, one object, one camera, no tangential distortion, correct matches, 6 seeds.
research/sensing-burstcap · 6 tests
Inverting the burst break-even formula: the largest sensor-area ratio a K-frame burst matches runs from √K in the dark to K in bright light, so 8 frames at 3 e⁻ read noise match a 4×-area frame from 4.5 e⁻ per pixel per frame, a 6× frame from 21 e⁻ and never an 8× frame; Monte Carlo agrees to 1% in variance at 18 settings. Model only, swept ranges not sourced to a device
Abstract
sensing-burst gave the break-even frame count N* = A2(e + r2)/(A e + r2) for a burst to match one frame from a sensor with A times the pixel area. This project solves it for A: the largest matched ratio is the positive root of (e + r2)A2 − K e A − K r2 = 0, between √K (read-noise limited) and K (shot-noise limited). Swept over 1 to 10,000 electrons per frame, read noise 1 to 10 electrons and K = 4, 8, 15: with 8 frames and 3 e− read noise a burst matches a 4× frame from 4.5 e− per pixel per frame and a 6× frame from 21 e−, and never an 8× frame; 15 frames match 12× from 3.6 e− (r = 1) to 358 e− (r = 10). A Monte Carlo of the Poisson-Gaussian model gives a variance ratio of 0.990 to 1.009 at 18 settings. Limits: it is the same model as sensing-burst (perfect registration, static scene, no clipping, dark current or fixed-pattern noise); the electron and read-noise ranges are swept, not taken from any device, because the one public sensor-data site found is all rights reserved.
research/sensing-calibviews · 4 tests
On OpenCV's 13 real checkerboard photos, random subsets of 3 views put the focal length within 2% of the 13-view value in 99% of draws (5–95% band 2.3% wide), 12 views give 0.5%; noise-only re-renders predict the band, and a one-term lens model doubles held-out error. One camera, one board
Abstract
Calibrates a camera with cv2.calibrateCamera on random subsets of the 13 real 640×480 checkerboard photos shipped with OpenCV (Apache-2.0), against the same model's all-view fit. With the default five-term model the 5–95% band of the focal length is 2.3% wide at 3 views and 0.49% at 12, and 99–100% of draws are within ±2%. Re-rendering the same board poses with Gaussian noise at the measured 0.41 px reprojection error and a known truth gives 3.0% and 0.8%, so sampling noise alone predicts the spread. A lean model (one radial term, centred principal point) has a 4.4% band on real photos against 2.0% on matching synthetic ones, and 0.43 px held-out reprojection error against 0.21 px for the full model at 12 views. Limits: one camera, one tutorial board with strongly tilted views; the reference is the all-view fit, not a lens ground truth; the 12-view row overlaps heavily (13 distinct subsets); synthetic noise is isotropic Gaussian.
research/sensing-epigate · 5 tests
An epipolar gate removes 93% of 10% random-pixel mismatches (5% of good observations lost) and takes 5% contamination from 0.168 to 0.0061 object radii, but a RANSAC start does as well at 10% and better at 20%; near-miss mismatches are mild for a robust loss already. Synthetic only
Abstract
Follows sensing-baoutlier with the geometric verification it left out. For each pair of nearby views a fundamental matrix is fitted by RANSAC from the matches alone (normalised 8-point, Sampson distance under 2 px, no use of the wrong poses) and an observation is kept if half its pairs accept it. On the 12-view rig it removes 94%, 93% and 89% of random-pixel mismatches at 5%, 10% and 20% while losing 2%, 5% and 16% of good observations; at 5% Cauchy adjustment improves from 0.168 to 0.0061 object radii (floor 0.0015). It is no better than the RANSAC pair start: equal at 10% (0.045) and worse at 20% (0.41 to 0.77 against 0.13). Near-miss (10 to 40 px) and along-epipolar-line mismatches are removed at 79–87% and are mild for a robust loss anyway (0.005 at 10%). Limits: synthetic, my own mismatch models, 6 seeds, untuned gate settings, scoring set differs from sensing-baoutlier.
research/sensing-gatecorr · 4 tests
When a bad feature is wrong by the same offset in more consecutive views, the gate's recall falls from 0.81 to 0.59 at a 10% wrong rate and its lead over a RANSAC start shrinks from 1.7x to a tie within seed noise, though typical errors stay at 2 to 3.5 times the clean floor; a single failed seed in 12 can dominate a mean
Abstract
sensing-gatemix drew every mismatch independently and flagged that as flattering the gate's voting. Here each bad feature is wrong by the same 10–40 px offset in span consecutive views (1, 2, 4, 8, 12 of a 12-view synthetic rig), at the wrong-match rates SIFT gives after a ratio test of 0.8, 0.7, 0.6 (10.2%, 6.1%, 4.7%, measured by sensing-matchreal). Gate recall falls from 0.81 to 0.59 at 0.8; gate + Cauchy adjustment goes from 0.0031 to 0.0048 object radii (clean floor 0.0014) over spans 1 to 8 while the RANSAC start stays at 0.0051–0.0062, so the gate's 1.7× lead becomes 1.06×, a tie at 12 seeds. One seed in 12 failed outright (0.64 for gate + Cauchy at span 12; 0.24 for plain Cauchy at span 1) and dominates those means. Limits: the span is swept, not measured; synthetic geometry; one image-space offset model, near-miss type only; 12 seeds.
research/sensing-gatemix · 4 tests
With the mismatch rates and mixes measured from SIFT on a real stereo pair, a ratio test of 0.8 to 0.6 (10% to 5% wrong) leaves gate + Cauchy adjustment at 0.008 to 0.003 object radii against 0.013 to 0.004 for the RANSAC start, and the all-random-pixel assumption overstated plain Cauchy's error 1.3 to 59 times; with no ratio test (56% wrong) every method fails
Abstract
sensing-epigate assumed random-pixel mismatches; sensing-matchreal measured what SIFT actually produces. Here the measured mismatch rate and the measured along-epipolar / near / far mix, per ratio-test threshold, drive the epigate bundle-adjustment trial (synthetic 12-view rig, 300 features, wrong poses, 8 seeds). At a ratio test of 0.8, 0.7 and 0.6 (10%, 6%, 5% wrong) gate + Cauchy adjustment gives 0.0082, 0.0032, 0.0027 object radii (clean floor 0.0014) against 0.0132, 0.0049, 0.0036 for the RANSAC pair start without a gate; at 0.9 (23% wrong) they tie (0.095 vs 0.087); with no ratio test (56% wrong) every method exceeds 0.6. The gate removes about 83–86% of the measured mix against 95–97% of uniform mismatches, because the along-epipolar share passes pairwise tests, yet the adjustment is harmless to those. Treating all mismatches as random pixels overstated the error of plain Cauchy adjustment from the all-view start by 1.3 to 59 times. Limits: the geometry is synthetic and only the rate and mix are measured, from one stereo pair and one detector; rate is applied per observation; two isolated cells (mutual check, ratio 0.9 and 0.6) show one-seed spikes in 8 seeds.
research/sensing-ghostedge · 6 tests
Does modelling the sub-pixel coverage of a moving subject fix the edge ring? Yes, on a known shape: a layered disc-over-background model fitted from the burst lifts the ring from 27.1 to 33.1 dB at 8 frames and 2 px per frame (4×-area frame: 31.7 dB), less at slow speeds, and it is sensitive to a 0.5 px centre error. Synthetic flat disc only.
Abstract
Follow-up to sensing-ghostring, where no merge or mask fixed the edge ring. Here each frame is a layered mix, a·disc + (1−a)·background, with sub-pixel coverage a from a fitted disc, and the background solved by least squares over the frames. On the same synthetic bursts at 8 frames and 2 px per frame the ring goes from 27.1 dB (aligned plus reject) to 33.1 dB with the disc fitted from the burst itself (large sensor, 4× area: 31.7 dB); at 0.5–1 px per frame it gains 0.4–2.2 dB and stays below the large frame. It is sensitive to geometry: a 0.5 px centre error drops the ring to 25 dB. Stylised: the model assumes the exact shape of the synthetic subject (a flat disc), so this is an upper-bound-style result, not a general deghosting method.
research/sensing-ghostring · 6 tests
Can a temporal-median background or a subject mask fix the edge ring that burst alignment leaves behind? No: four ring treatments leave it at 21.6–27.2 dB at 8 frames against 31.6–32.0 for a 4×-area frame, none beats plain aligned-plus-reject (27.0 dB), and the mask misses most of a slow subject. Synthetic scenes only.
Abstract
Follow-up to sensing-ghostalign, on the same synthetic bursts (one disc moving across a still scene, Poisson shot noise, Gaussian read noise, a large sensor with 4× the pixel area). Four explicit treatments of the edge ring — frame 0 alone, a per-pixel temporal median, the aligned merge, the reject-only merge, each gated by a subject mask estimated against the temporal median — all leave the ring at 21.6–27.2 dB at 8 frames against 31.6–32.0 dB for the large frame. None beats plain aligned-plus-reject (27.0 dB at 1 px per frame); the median is the worst (22.3 dB) unless the subject moves 4 px per frame, and the mask recalls only 44–64% of a slow subject. A negative result. Stylised: synthetic scenes, one flat rigid disc, translation only.
research/sensing-kploc · 5 tests
On 30 warped real photographs SIFT keypoints localise to 0.13 px per axis at zero noise and 0.32 px at noise sigma 10 grey levels, 0.59 px at 40, so sensing-ba's assumed 0.5 px sits at roughly sigma 28; large keypoints are worse, blur matters little
Abstract
sensing-ba and sensing-recon assumed 0.5 to 1 px of Gaussian pixel noise on every observation. This project measures it: OpenCV SIFT on ten scikit-image photographs, each warped by three random known homographies, with Gaussian noise (0 to 40 grey levels) or blur (0 to 3 px) added to both images, and the error of every ratio-passing mutual match that lands within 3 px of the truth. Per keypoint and axis, the localisation sigma is 0.13 px with no added noise (partly bilinear resampling: an integer shift gives exactly zero), 0.23 at sigma 5, 0.32 at 10, 0.43 at 20 and 0.59 at 40; through sensing-recon’s 0.0028 radii per px that is a surface error of 0.0004 to 0.0017 radii. So 0.5 px is a pessimistic input for a well-exposed image and about right for a very noisy one. Large keypoints (diameter over 16 px) are 4× worse than small ones when clean; blur of 3 px costs 1.5×. Limits: Gaussian noise rather than shot noise, synthetic warps, one detector, errors truncated at 3 px (a lower bound, 77% kept at sigma 40), not fed back into the reconstruction.
research/sensing-matchmix · 3 tests
With the mismatch rates and along/near/far mixes measured for SIFT, a ratio test of 0.8 or tighter (10% wrong or fewer) leaves gate and RANSAC-start adjustment near the clean floor (0.002 to 0.02 radii) while an all-view start is 0.005 to 0.39; at ratio 0.9 the RANSAC start wins and with no ratio test (56% wrong) everything fails. Synthetic rig, measured mixes
Abstract
Takes the next step of sensing-matchreal: its measured ratio-test mismatch mixes (rate and the share along the epipolar line, near and far) are injected into the sensing-epigate rig of 12 views with wrong poses. At ratio 0.7 (6% wrong, 62% along) Cauchy-loss adjustment from an all-view start gives 0.052 object radii, a RANSAC start 0.0060, an epipolar gate 0.0038 and gate plus RANSAC start 0.0036, against a clean floor of 0.0014. At ratio 0.9 (23% wrong) the RANSAC start (0.088) beats the gate (0.17); with no ratio test (56% wrong on stereo, 40% on warped photographs) every method is above 0.4. The gate beats the RANSAC start up to 20% contamination at the ratio-0.7 shares and fails at 30%. The mismatch statistics are measured; the scene, poses and Gaussian noise are the synthetic rig's, mismatches are independent per observation, and there are 6 seeds with no intervals.
research/sensing-matchreal · 4 tests
On a real Middlebury stereo pair SIFT with no ratio test is 56% wrong and 84% of the wrong ones are far, but at a 0.7 ratio test 6% are wrong and 62% of those lie on the true epipolar line (chance: 1%); ten warped photographs give 40% wrong without and 2% with ratio 0.6
Abstract
sensing-epigate injected mismatches of three invented kinds. This project measures them: OpenCV SIFT on the Middlebury 2014 Motorcycle stereo pair (truth from its disparity map) and on ten scikit-image photographs warped by known homographies, with every wrong match classed as along-epipolar, near (within 40 px) or far. With no ratio test 56% of stereo matches are wrong, 84% of those far; Lowe's ratio test removes the random-looking ones first, so at 0.7 only 6% are wrong but 62% of those lie within 2 px of the true epipolar line (random keypoints land there 1% of the time), the case a single-pair epipolar test cannot reject. Warped photographs: 40% wrong without a ratio test, 2% at 0.6, survivors 12–19% near. Limits: one stereo pair and one detector, ground truth good to about a pixel, class boundaries are mine, not fed back into bundle adjustment.
research/sensing-mixgate · 4 tests
Feeding sensing-matchreal's measured SIFT mismatch mixes into the sensing-epigate trial, gate plus Cauchy beats a RANSAC start at ratio tests of 0.8 and tighter (0.0078 against 0.0207 object radii at 0.8) and loses only at a loose 0.9 ratio test, where all methods exceed 0.08; synthetic geometry, 6 seeds
Abstract
sensing-epigate found a fundamental-matrix gate no better than a RANSAC start against invented mismatches; sensing-matchreal then measured real SIFT mismatches on a Middlebury stereo pair. This project runs the epigate trial with each measured mix (rate and along/near/far shares at ratio tests none, 0.9, 0.8, 0.7, 0.6). Gate plus Cauchy adjustment gives 0.0078, 0.0033 and 0.0026 object radii at ratio 0.8, 0.7 and 0.6, against 0.0207, 0.0058 and 0.0032 for a RANSAC start and 0.43, 0.013 and 0.0057 for plain Cauchy (floor 0.0015). At ratio 0.9 (23% wrong) the RANSAC start is better (0.082 against 0.130) and with no ratio test (56% wrong) nothing works. An all-uniform control at the same rates is dominated by single failed seeds and is reported as unestablished. Limits: one scene and detector for the mix, synthetic orbit geometry, independent draws, 6 seeds, no intervals, untuned gate.
research/sensing-nees · 6 tests
With a 1/f accelerometer bias, a filter with no bias state claims 2.9 m after a 300 s GNSS outage when the RMS is 151 m (NEES/dof 514 to 2,000 at 100 ug); a random walk read off the Allan curve at 1 s or 10 s claims 2.4 to 3.7 times the true error, and at 100 s close to honest (0.9 to 1.6). Synthetic bias, swept instability
Abstract
Tests whether a 1-D INS/GNSS Kalman filter reports its own error correctly when the accelerometer bias is 1/f noise (five Gauss-Markov processes, 1 to 10,000 s) on top of white noise of 70 µg/√Hz (ICM-42688-P product page). Bias instability is swept from 10 to 300 µg and GNSS drops out for 300 s. NEES per degree of freedom (1 is honest): a filter with no bias state scores 21 to 21,700; a random walk read off the Allan curve at 1 s or 10 s scores 0.06 to 0.16 at 300 s (too pessimistic) and at 100 s 0.9 to 2.3; the matched model scores 0.88 to 1.05. At 100 µg the 300 s position RMS is 66 m matched, 81 m tuned at 100 s and 103 m tuned at 10 s. The bias is synthetic with a swept (not sourced) instability, one axis, no attitude coupling, 400 Monte Carlo paths.
research/sensing-viomarg · 5 tests
A Schur-complement prior in the sensing-vio planar smoother cuts rms drift on 30 s paths from 12.3 % to 2.5 % of path (8 seeds, baseline IMU), and IMU grade then orders the results (0.25x noise 2.1 %, 1x 2.5 %, 4x 6.5 %), unlike with the pinned prior; planar and synthetic, not EuRoC
Abstract
sensing-vio found that on 12 s paths only camera resolution moved drift, and that its smoother diverged on longer paths because the oldest pose was pinned and old velocity and bias information was forgotten. This project replaces the pin with a Schur-complement prior: after each solve, the oldest keyframe is marginalised out of its prior and the IMU factor to the next keyframe, giving a dense 8-state prior on the next one (checked against a brute-force profile in a test; the dropped keyframe's bearing factors are not marginalised). Same world, sensors and factors, 8 paired seeds, 30 s paths (about 30 m). Pinned prior: rms drift 12.3 ± 5.3 % of path at baseline IMU, 23.6 ± 7.4 % at 0.25× IMU noise, 12.3 ± 5.5 % at 4×, with worst seeds 46 to 63 %. Marginalised: 2.5 ± 0.2 % at baseline (paired ratio 0.32), 2.1 ± 1.2 % at 0.25× (median 0.8 %, one seed at 10 %), 6.5 ± 0.9 % at 4×. So with marginalisation a better IMU helps and a worse one hurts, which the pinned smoother hid. Limits: planar, point landmarks, known association, illustrative noise levels, 8 seeds, solver capped at 15 iterations, one path length, no real dataset. Noise densities are the sensing-vio values (of the order of EuRoC's published figures), not fitted.
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.