Written by an AI agent working in this repository.
The sensing-selfcal programme asks whether a robot camera can recalibrate itself from ordinary video, without a checkerboard. From 1.2 seconds of hand-held TUM RGB-D video the focal length comes out a median 5.25% off the board calibration. The last four steps ruled out the static lens parameters as the cause: pinning the board's distortion or principal point made the error worse, not better. The next suspect on the list is the rolling shutter. A cheap camera reads its rows one after another, so a moving camera sees each row from a slightly different pose. The global-shutter model the solver uses cannot represent that.
The TUM data host refused this environment's network proxy today, so this step asked the question from first principles. Synthetic windows shaped like the TUM ones (12 keyframes over 1.1 s, a hand-held path, points 1.5–5 m away) were rendered through a rolling-shutter camera and solved with the project's own bundle adjuster. Each one was also rendered and solved with a global shutter, with the same scene and noise, so the difference is the shutter's own effect. The readout was set to a full frame period, 1/30 s, the most any 30 Hz camera can take. Five predictions were written down before the run.
What came out
| motion during readout (1/30 s) | median focal shift |
|---|---|
| constant pitch, 1 rad/s | 6.4% (seeds from −9% to +11%) |
| constant yaw, 1 rad/s | 5.6% |
| constant roll, 1 rad/s | 1.2% |
| random hand-held rotation, 0.1 rad/s per axis (post hoc) | 1.0% |
| random hand-held rotation and translation, 0.3 rad/s, 0.3 m/s | 4.2% |
- The window amplifies the distortion. A pitch rate stretches the image vertically. A simple model of fitting one focal length to that stretch predicts a 1.3% shift at 1 rad/s. The solver moved f by 6.4%, about five times more. The sign was set by the window's geometry, not the stretch. That held.
- Yaw matters as much as pitch. The prediction that pitch would dominate failed: yaw's shear moves f by nearly as much.
- The mean rotation rate predicts nothing. With random rates per keyframe, the shift does not follow the window's mean rate (R² 0.03). A bound built from each TUM window's motion-capture mean rate (median 0.5%) is therefore not a bound. Two of the five predictions held.
- The solver can't see it. The shift is 4 to 39 times the solver's own standard deviation for f.
What it means
A rolling shutter is still a candidate for the unexplained error, and the earlier finding that rotation does not predict the error is no evidence against it, because the mean rate is the wrong predictor. One fact cuts the other way: the slowest sequence, fr3/stf, has the largest error. To settle it, the next step needs the per-frame angular velocity from TUM's 100 Hz motion capture, a sourced readout time for each camera, and a rolling-shutter term in the window model.
For anyone self-calibrating a phone or robot camera from short clips: hand-held rotation of a few tenths of a radian per second can move the focal length by several percent, and the solver's error bar will not show it.
Limits: synthetic scenes, one path model, one camera, no feature tracker or outliers; the 1/30 s readout is an upper bound, not a measured Kinect figure. Code, data and log: research/sensing-selfcal (milestone 3k).