← All research topicsGitHub
Research topic

Autonomous multi-angle capture

Filming a travel creator from every angle with open, packable hardware and an automated edit: the pixel, perspective, battery and weight limits first, in simulation.

34 projects, each with its own page, code and tests. See them on the full research index.

How long a lens can a 360 camera fake by cropping into a reel, and what perspective can no crop fake?

research/capture-crop · 10 tests

How long a lens can a 360 camera fake by cropping into a reel? An 8K equirect is native only to ~24 mm-equivalent at 1080x1920 (85 mm is a 3.5x upscale), and perspective is set by distance: a stick at 0.8 m enlarges a 10 cm-nearer feature 14% vs 7.6% at an 85 mm framing distance, so portrait looks need a camera standing back. Pixel-count upper bound. First project of track 13.

Abstract

Pixel and perspective geometry for the simplest multi-angle rig, a 360 camera on a pole reframed in software. An 8K (7,680 px) equirect cropped into a 1080×1920 vertical reel is native only up to about 24 mm full-frame-equivalent; 50 mm is upscaled 2.1× and 85 mm 3.5×, and a native 85 mm crop would need a ~27,000 px equirect. Perspective is set by distance: from a stick at 0.8 m a feature 10 cm nearer the lens images 14% larger, against 7.6% from the 1.4 m an 85 mm head-and-shoulders shot is taken at, with the background 1.7× larger. So portrait and compressed-background shots need a camera standing back, whatever the resolution. Upper bounds from pixel counts (real cameras resolve less); framing distances and depths are stated assumptions.

What should a travel creator pack to film from many angles under a weight budget and the 100 Wh battery limit?

research/capture-kit · 10 tests

What should a travel creator pack to film from many angles within a weight budget and the 100 Wh battery limit? Exact search over 17,280 kits reaches all 18 band x viewpoint shots at 5 kg; a greedy packer stalls at 12 because lenses need their body (assumed device figures).

Abstract

Exact search over 17,280 kits of cameras, lenses, mounts and battery packs (11,520 legal), scoring distinct focal-band × viewpoint shots (18 at most) and recording hours against carried weight. With a 4 h recording floor, exact search reaches 15 shots at 4 kg and all 18 at 5 kg, while a greedy best-gain-per-gram packer stalls at 12 from 3 kg because a lens is worthless without its body. Without a floor, variety-first search returns kits that record only ~2.2 h. Every device figure is assumed (round class-typical numbers, not datasheets); only the 100 Wh limit is sourced.

How many cameras, and where, to film a person from every angle?

research/capture-rig · 8 tests

How many cameras, and where, to film a person from every angle? A 30° gap between neighbours needs 13 cameras on a ring (15° needs 25), but depth error is set by convergence angle, not count: a pair's error falls 11× from 5° to 90° (7.3 to 0.64 mm) while 3 vs 16 ring cameras only halves it. Idealised 2-D lower bounds.

Abstract

A 2-D geometric model of N cameras on a ring or a 90° front arc 3 m from a subject, scored on angle coverage and triangulation error. Cameras buy coverage, angle buys depth accuracy: a 30° gap between neighbours needs 13 cameras (15° needs 25), while centre depth error barely moves with count (0.52 mm for 3 on a ring, 0.23 mm for 16) but falls 11× as a pair’s convergence angle goes from 5° to 90° (7.3 mm to 0.64 mm). Four cameras on a front arc are within 11% of four on a ring. Idealised lower bounds with assumed noise (0.5 px) and no occlusion; novel-view quality is not modelled.

How far apart can cameras be before a reel editor can no longer synthesise the angle between them?

research/capture-novelview · 7 tests

How far apart can cameras be before a reel editor can no longer synthesise the angle between them? With exact depth a two-view warp beats cutting to the nearest camera only up to about 30° (24.8 vs 23.6 dB); beyond that holes dominate, and a bystander pulls the limit in to about 15° (ray-traced toy scene; ordering, not decibels).

Abstract

A ray-traced toy scene, two ring cameras, a depth-based warp to the midpoint, scored against the ray-traced truth. With exact depth, synthesis beats cutting to the nearest camera only up to a gap of about 30° (24.8 against 23.6 dB); at 45° it is worse (19.3 against 23.1 dB) because 8% of the target is a hole neither camera saw, though pixels it does fill stay near 29–30 dB. A bystander pulls the crossover in to about 15°, and depth error on the sources costs about 1.4 dB per percent at first (27.5 dB exact, 22.0 dB at 10%). Toy scene with smooth textures and oracle depth: read the ordering, not the decibels.

Is the depth a camera rig can actually measure good enough for the in-between angle?

research/capture-stereodepth · 7 tests

Is the depth a camera rig can actually measure good enough for the in-between angle? Block-matching stereo on rendered pairs feeds the two-view warp: mean depth error 14.9% at 3 cm baseline to 2.7% at 50 cm, coverage peaks at 91%, and the result never beats cutting to the nearest camera (22.3 vs 24.8 dB at a 15° gap; exact depth 27.3 dB).

Abstract

capture-novelview assumed independent depth noise of 2–5%. Here depth is measured instead: block-matching stereo on a rendered rectified pair, fed into the same two-view warp and scored against ray-traced truth at a 15° gap. Mean depth error falls from 14.9% at a 3 cm baseline to 2.7% at 50 cm, but coverage peaks at 91% near 20 cm (stereo cannot match occluded or flat pixels), and the synthesised view never beats cutting to the nearest camera: best 22.3 dB at 20 cm against 24.8 dB, with exact depth at 27.3 dB. Independent noise with the same mean error is optimistic by 1optimistic by 1–5 dBndash;7 dB, because real errors are correlated and missing depth leaves holes. Sensor noise of 3% already costs 3 dB. Toy scene, smooth textures, a classic matcher with no learned prior or hole filler.

Can the holes and correlated errors of measured stereo depth be repaired so the in-between view beats cutting to the nearest camera?

research/capture-hybrid · 6 tests

Can the holes and correlated errors of measured stereo depth be repaired so in-between synthesis beats cutting to the nearest camera? A hole filler and a no-ground-truth disagreement gate that falls back to the raw nearest camera lift a 15° gap from 22.3 dB to 29.2 dB against 24.8 dB for cutting (held-out seeds 29.2); with a bystander the gate loses its edge (26.1 vs 26.3 for plain filling). Toy scene.

Abstract

capture-stereodepth found that synthesis from block-matching stereo depth never beat cutting to the nearest camera (22.3 against 24.8 dB at a 15° gap). Here the same holey, correlated depth is repaired two ways: filling holes with the farther of the nearest valid row neighbours (27.4 dB), and a gate that needs no ground truth, where the two warped sources disagree or a hole remains, falls back to the raw nearest camera (29.2 dB, 4.4 dB above cutting; the same on three held-out noise seeds). Gains hold from 8° to 30° gaps and at 6–50 cm baselines, shrink to nothing at 6% sensor noise, and with a bystander the gate loses to plain hole filling (26.1 against 26.3 dB). A per-pixel oracle at 32.0 dB bounds what any confidence rule could reach. Stylised toy scene; read the ordering, not the decibels.

Which platform can follow a travel creator through a crowd and keep the shot?

research/capture-follow · 10 tests

Which platform can follow a creator through a crowd and keep the shot? A drone that ignores pedestrians frames 99.9% of the time but falls to 39% at 0.3 people/m² once it avoids them; a wheeled robot is stopped by step lines 42–47% of the walk, and a flat-floor one that ignores people passes within 0.5 m ~500 times an hour. 2-D simulation, assumed platform figures.

Abstract

A 2-D crowd simulation of a tripod, a wheeled robot, a wheel-legged robot and a drone holding a camera 2.5 m behind a walking creator among pedestrians who do not react to it. A drone that ignores pedestrians frames the creator 99.9% of the time at any density, but steering around people drops it to 69% at 0.1 people/m2 and 39% at 0.3; a ground robot on a plaza with curbs and one ramp is stopped 42–47% of the walk, and wheel-legs hold the frame 70% of the time in a thin crowd but 4.5% in a dense one. Avoidance and framing trade off: a flat-floor wheeled robot that ignores people frames 98% of the time while passing within 0.5 m of a pedestrian about 500 times an hour. All platform speeds, powers and masses are assumed; pedestrians do not avoid the platforms, and the drone is not limited by altitude, wind or regulation.

How much of a multi-camera shoot can an automatic edit recover, and what stops it?

research/capture-edit · 9 tests

How much of a multi-camera shoot can an automatic edit recover? At detector noise 0.3 a fixed 3 s rotation shows mean true quality 0.546, a cut-cost (Viterbi) edit 0.802 vs 0.815 on true scores, and best-score-every-frame 0.723 at 345 cuts a minute; clock sync fails first on short clips (5 s at noise 0.25: 25% over 40 ms). Synthetic quality model, no real footage.

Abstract

A synthetic model of four cameras whose true angle quality changes every 2–8 s, seen through a noisy detector. At noise 0.3 a fixed 3 s rotation shows mean true quality 0.546; an offline Viterbi edit that maximises detected quality minus a cost per cut reaches 0.802 against 0.815 for the same search on true scores, while picking the best score every frame gives 0.723 with 345 cuts a minute and every shot under 1.5 s. The cut cost matters more than detector accuracy (best near 1.0, about 14 cuts a minute). Clock sync from a shared loudness envelope fails first on short clips: a 5 s clip at noise 0.25 is off by over 40 ms in 25% of trials, 15 s in 5%. No real footage, reframing or film-look grade; quality is a made-up random process.

When does automatic blurring miss a bystander's face, and which filming platform is conspicuous?

research/capture-bystander · 11 tests

When does automatic blurring miss a bystander's face? On an assumed size-recall model a face detector leaves 5% of identifiable faces unblurred per frame, blurring the head of any detected body under 0.1%, and independent per-frame misses show a face in some frame of a 5 s shot 100% of the time; an offline whole-track blur leaves 0 to 5%. No real footage or detector.

Abstract

An assumed size-to-recall model of blurring strangers' faces while filming in public. A face detector that is 50% reliable at 20 px leaves a face-on face of at least 32 px unblurred 5% of the time per frame (8% at 5 m on an 8K 360 camera); blurring the head region of any detected body instead drops it under 0.1%, because a body is 11× taller than a face. A telephoto frame resolves about 10 identifiable people at once against 2.5 for an 8K 360 view. Per-frame blur is worse than it looks: with independent misses a person filmed for 5 s at the edge of range shows unblurred in some frame 100% of the time, 25% if misses last the whole shot, and an offline edit that blurs a whole track after one detection is left with 0–5%. Size alone does not separate the tall platforms. All detector curves, sizes and angles are assumptions; no real detector or footage; blurring is not legal compliance.

Does recording only when something happens save a travel creator's battery, and what does it cost in missed moments?

research/capture-trigger · 7 tests

Does recording only on a trigger save battery, and what does it miss? With 6 thirty-second moments an hour, standby with a 10 s pre-roll uses 19.4 Wh of an 80 Wh continuous day (76% less) and captures 90% of interesting seconds; deep sleep uses 7.7 Wh but captures 69%. Misses and wake latency, not the battery, set coverage. Assumed device figures, synthetic day.

Abstract

A synthetic 16 h day with assumed device figures. With 6 interesting 30 s moments an hour and a trigger that misses 10% and raises 2 false alarms an hour, a standby camera with a 10 s pre-roll buffer uses 19.4 Wh against 80 Wh continuous (76% less) and captures 90% of the interesting seconds; a deep-sleep camera uses 7.7 Wh but captures 69%, because it wakes too slowly to catch the start. Coverage is set by the trigger's miss rate and the wake latency, not the battery; energy decides only for a rig (four cameras: 320 Wh continuous, 78 Wh triggered). Stylised: every device figure is assumed and the trigger is a probability, not a detector.

When a camera rig records only on a trigger, should every camera have its own, or should one wake the rest?

research/capture-wake · 9 tests

When a camera rig records only on a trigger, should every camera have its own, or should one wake the rest? Four standby cameras with their own triggers use 80 Wh of 320 Wh continuous but get all four angles on only 68% of interesting seconds; one sentinel waking the others costs the same and gives 90.5%, and any-camera wake gives 100% for 92 Wh but its false-alarm energy grows with detector count. Assumed device figures.

Abstract

A synthetic 16 h day with assumed device figures and four standby cameras. With 6 thirty-second moments an hour and a 10% miss per detector, a trigger per camera uses 80 Wh (of 320 Wh continuous) but puts all four angles on only 68% of the interesting seconds, since independent misses differ by camera (≈0.94). One sentinel waking the rest costs the same 80 Wh and gives 90.5%; waking all on any camera's trigger gives 100% for 92 Wh but its false-alarm energy grows with the number of detectors (281 against 126 Wh at 40 false alarms an hour). Sleeping the other cameras halves the rig to 41.5 Wh at the cost of the first 7 s of each moment. If the misses are shared, extra detectors buy nothing. Stylised: every figure assumed, not measured.

Where in a camera ring should the sentinel go when the subject faces a favourite direction?

research/capture-sentinel · 13 tests

Where in a camera ring should the sentinel go when the subject faces a favourite direction? A sentinel facing the favourite direction misses 8.6% of moments against 34.4% for the one behind; spread cameras' misses are anti-correlated, so capture-wake's independence assumption overstates the all-miss rate by 2x on a ring of 4 and understates it 12x for a tight cluster. With the turned-face miss measured in capture-detectors (54 faces) in place of the assumed curve, placement matters only if the back of a head goes undetected (untested), and weak detectors miss 8 to 45% of moments even with four cameras listening.

Abstract

A model with assumed detector figures (a miss rate of 3% when the subject faces the camera, 40% when facing away) and a subject whose facing at each moment is von Mises distributed around a favourite direction. In a four-camera ring with concentration κ = 2, the sentinel at the favourite direction misses 8.6% of moments, the one behind the subject 34.4%, and a rotating sentinel 21.5%; the closed form matches a 40,000-moment simulation to within 0.4 points. Misses across a spread ring are negatively correlated (a subject turned from one camera faces another), so assuming independent misses, as capture-wake did, overstates the chance that every detector misses by about 2× on a ring of four; for a tight cluster of cameras the correlation is positive and the independent figure understates it about 12×. A second part replaces the assumed cosine curve with the miss measured on 54 faces in capture-detectors (YuNet at 0.6: 3% near-frontal, 5% turned; YuNet at 0.9: 46% and 84%; frontal Haar cascade: 34% and 84%): placement then matters only if the back of a head goes undetected, which is untested (best 6–7% against worst 81% for YuNet at 0.6 if so, 3–4% against 5% if not), and for a weak detector the four-camera any-of miss is 0.08–0.45 rather than 0.0007. The cosine curve and the 54-face sample (a detector’s own detections, a landmark proxy for turn) are the limits. Stylised: the cosine figures are assumed; the step figures are measured on 54 faces.

Does a longer window, a lower gamma or a delay prior fix the period aliasing and early-start failures of GCC audio sync on real clips?

research/capture-aliaswin · 3 tests

Does a longer window or a delay prior fix GCC sync's aliasing on real clips? Window length (shared audio) is the lever: 97% wrong at 1 s, 38% at 2 s, 16% at 3 s on the failing clips; a tone clip needs a prior tighter than its period.

Abstract

Follow-up to capture-residgate section D. On ten real LibreOffice gallery clips (0 dB white noise, 0.8 s offset, 432 windows per clip and cell), the share of delay estimates over 2 ms off on the four failing clips falls from 97% at a 1 s window to 38% at 2 s and 16% at 3 s (gamma 0.25), because two cameras with a 0.8 s offset share only W−0.8 s of audio. Gamma moves 2 s results between 38% and 76% (PHAT worst). A tone clip (space2) is not helped by more audio (52–70% aliased at 2 and 3 s); only a delay prior tighter than its 12 ms period works (oracle ±10 ms: 0% wrong; ±50 ms: 65% still aliased). The early-start failures of untie and soft (59–77% wrong when the window starts in the clip's first half second, 0% later) are partly a short shared segment; the shared audio is only 2–5 dB quieter in wrong windows, so loudness is a weak explanation. Limits: ten clips, white noise at one SNR, 3 s windows barely vary their start (one draw of content per clip), the prior is an oracle ceiling.

How long a window of sound does it take to sync two drifting cameras, and how often must you redo it?

research/capture-audiosync · 6 tests

How long a window of sound syncs two drifting cameras? GCC-PHAT plus a robust line fit syncs to under 0.02 ms over an hour from -10 dB SNR with 2 s windows; a sync made once at the start drifts one frame in 833 s at 40 ppm.

Abstract

Synthetic speech-like audio (shared source, independent noise, optional reverb), per-window GCC-PHAT delay (100–4000 Hz) and a Theil–Sen line over 12 windows in 1 h at 40 ppm clock error. With 2 s windows the fit is below 0.02 ms from −10 dB SNR up and fails completely at −15 dB; 5 s windows hold to −15 dB. A sync made once at the start drifts one 30 fps frame (33.3 ms) in 833 s at 40 ppm (144 ms after an hour; analytic ppm × time reproduced exactly). Reverb at −10 dB direct-to-reverberant adds 4adds 4–8 msndash;5 ms (median). Not from real recordings: the audio and the exact delay are synthetic and the ppm values are swept, none sourced (no datasheet reachable).

How small can a face or a person be before a real detector stops finding it?

research/capture-detectors · 10 tests

How small can a face be before a real detector misses it? YuNet found every face of 16 px or wider and half at 10 px (assumed: 20), cutting the modelled blur leak on an 8K 360 camera from 4.8% to 0.01%; YOLOX finds half of people at 23 px tall. A pilot on 22 easy faces in 128 COCO photos: an optimistic bound.

Abstract

Measured recall against pixel size for real detectors on COCO photos, replacing the assumed curve in capture-bystander. 22 confident faces and 254 labelled people from 128 photos were shrunk by scales from 1 to 0.065 and each detector asked to find them again. YuNet at its usual 0.6 score found every face of 16 px or wider and half at 10 px (fitted 50% point 9.2 px, range 8.2–10.4), not the assumed 20; at a 0.9 threshold it is 26.6 px. On an 8K 360 camera the per-frame leak among faces of at least 32 px falls from the assumed 4.8% to 0.01% (36% to 0.9% at 16 px), and rises to 7.2% at the 0.9 threshold. A YOLOX-s person detector reaches 52% at 20–32 px of height and 94% at 96–128 px (50% point 23 px, not 40); the head sits in the top quarter to third of its box. Section F: by how far the face is turned from the camera, YuNet at 0.9 falls from 54% to 16% recall (54 faces) and the frontal Haar cascade from 66% to 16%, while YuNet at 0.6 holds 95%+; the turn is a landmark proxy and back-of-head is not covered. A pilot: the reference faces are easy by construction (frontal, confident), the photos are shrunk not distant, and n is small, so these are optimistic bounds; the identifiable size is still assumed.

Does a predictive gap-seeking planner recover the framing a follow camera loses when it avoids pedestrians?

research/capture-followplan · 5 tests

Does a predictive planner recover the framing a follow camera loses by avoiding pedestrians? Not at equal settings (drone 58% framed vs 68% for repulsion at 0.1 people/m²), but it halves-to-tenths the contacts (36 vs 161 per hour; wheeled 21 vs 326) and moves the framing/contacts front left. 2-D simulation, assumed platform figures.

Abstract

capture-follow found that a repulsion-field controller trades framing for distance: at 0.1 people/m² a drone that ignores pedestrians frames 99.8% of the time with 523 contacts (within 0.5 m) per hour, and 67.8% with 161 once it avoids them. Here a sampler that predicts pedestrians 3 s ahead and picks a velocity that goes round and rejoins is run on the same crowds (8 seeds). Framing is not recovered at equal settings: the planner at a 1.5 m radius frames the drone 58.0% of the time (repulsion 67.8%) but with 36 contacts an hour, not 161. What it changes is the trade-off: on the drone’s framing-against-contacts curve the planner sits to the left of repulsion (89% framed at 281 contacts against 79% at 322; 58% at 36 against 58.5% at 92), and on ground platforms at 0.1 people/m² it cuts contacts from 326 to 21 (wheeled, flat plaza) with the same 40% framing. At 0.3 people/m² both controllers lose the frame (3 to 39%). Pedestrians ignore the platform and their velocities are known exactly, so the planner is an upper bound on perception. Platform figures are capture-follow’s assumed round numbers, not datasheets.

How much of a predictive follow camera's contact reduction survives noisy, delayed pedestrian detections?

research/capture-followtrack · 4 tests

How much of a predictive follow camera's contact reduction survives noisy, delayed pedestrian detections? Framing is unchanged (57-61%) but a drone's contacts rise from 36 per hour with exact data to 120 at 0.3 m noise and 0.25 s delay, and 373 at 1 m and 1 s (ignoring pedestrians: 523). 2-D simulation, swept noise, perfect association assumed.

Abstract

capture-followplan's planner cut a drone's close passes (within 0.5 m) at 0.1 people/m² from 523 to 36 per hour, but it was handed exact pedestrian positions and velocities. Here a constant-velocity Kalman tracker over detections with Gaussian noise (0.1 to 1.0 m) and latency (0 to 1 s) feeds it instead, on the same 8 crowds. Framing is unchanged (57 to 61%, against 58.0% with exact data); contacts are not: 45 at 0.1 m noise and no delay, 120 at 0.3 m and 0.25 s, 199 to 257 at 0.6 m, 373 at 1.0 m and 1 s. The planner beats exact-data repulsion (161) only while noise is about 0.3 m or less and latency under about 0.5 s, and that comparison favours repulsion, which was not rerun with noisy inputs. Assumed, not measured: perfect detection-to-track association, independent Gaussian noise, the filter's process noise, and capture-follow’s round-number platform figures; the noise levels are a sweep, not a detector’s datasheet.

What does it cost a travel filming kit to also carry its footage backup?

research/capture-kitstore · 11 tests

What does it cost a travel filming kit to carry its footage backup? Counting storage costs 3 of 18 shots at 4 kg; the lightest kit reaching all 18 grows from 3,900 g to 4,125 g.

Abstract

Chooses the capture kit (11,520 kits of capture-realkit, sourced devices) together with the lightest nightly-offload storage plan from capture-storage (microSD cards, two SSD copies of a 7-day trip, parallel readers, copy within 1 h) under one weight budget. Counting storage costs 3 of 18 shots at a 4 kg budget (18 become 15), and the lightest kit reaching all 18 shots grows from 3,900 g to 4,125 g (+225 g). Scaling bitrates from 0.25× to 1.5× moves that total between 4,025 and 4,300 g, and reader speed (90, 104, 312 MB/s) changes no shot count at any budget tried. Sweeping filming hours (2–8 h/day) and trip length (3–28 days) moves the lightest 18-shot total from 3,492 g to 5,307 g; within 4 kg only the 2 h/day kit keeps all 18 shots. Sourced: the action-camera bitrate, microSD mass, bus ceilings and the device catalogue. SSD and reader masses were checked against retailer listings (32–58 g per SSD, 28–85 g reader): the lightest 18-shot kit then weighs 4,040–4,262 g and 4 kg still reaches 15 shots. Not sourced: the 360-camera, phone and mirrorless bitrates (swept), and the laptop or hub the offload needs (not counted).

Can a small model trained on many random scenes choose the per-pixel blend weight for stereo-depth view synthesis better than a hand-set ramp?

research/capture-learngate · 7 tests

Can a small model trained on many random scenes pick the per-pixel blend weight better than a hand-set ramp? A one-hidden-layer network beats the best soft gate by 0.57 dB on 40 held-out scenes (all 40 ahead) and recovers 0.36 of the headroom to the best per-pixel blend.

Abstract

A one-hidden-layer network that predicts, per pixel, how much of the raw nearest camera to blend into a stereo-depth synthesis scores 26.78 dB on 40 held-out scene configurations (new ground textures) against 26.21 dB for the best soft disagreement gate and 25.81 dB for the hard gate, ahead of the soft gate in all 40 (+0.57 dB, standard error 0.05). That recovers 0.36 of the 2.7 dB between the hard gate and the best possible per-pixel blend (soft: 0.15; ridge regression: 0.29). Outside the training ranges (gaps of 36–46°, three bystanders, heavy noise) the gain over the soft gate falls to +0.24 dB, and the matching cost adds almost nothing as a feature. Toy analytic scene, block matcher, PSNR only; every scene parameter is drawn from stated ranges, which are assumed, not sourced.

Does the audio-sync threshold found on synthetic noise hold on real recorded sound?

research/capture-realaudio · 7 tests

Does the synthetic audio-sync threshold hold on real sound? No: on 10 real clips 2 s windows fail 53% of the time at -10 dB SNR (0% synthetic), and partial whitening instead of PHAT halves that.

Abstract

Reruns capture-audiosync's GCC-PHAT delay estimator on 10 real recordings and sound effects (installed LibreOffice gallery sounds, not copied into the repository) with simulated delay and synthetic white or wind-like noise. With 2 s windows the share of windows more than one 30 fps frame off is 0.00 on synthetic audio at −10 dB SNR but 0.53 on average over the real clips (0.00 for applause, 0.95 for the worst), and 0.26 vs 0.00 at 0 dB. Partial whitening (dividing the cross-spectrum by |C|γ) cuts the mean failure share from 0.52 to 0.28 at −10 dB with γ = 0.25. Limits: 10 sound-effect clips, not field audio; simulated delay and noise; no reverb, where PHAT is expected to win; trials within a clip overlap.

Does an automatic multi-camera edit survive the audio-sync errors measured on real recorded clips?

research/capture-realedit · 6 tests

Does an automatic multi-camera edit survive sync errors measured on real clips? Not as on synthetic audio: 34% of cuts jump over a frame at 0 dB SNR (64% at -10 dB) against none, depending on the clip; partial whitening cuts the 0 dB figure to 1.4%.

Abstract

Reruns capture-syncedit with every GCC window computed on 10 real recorded clips (the LibreOffice sound gallery, via capture-realaudio) instead of synthetic audio. 4 cameras, a 120 s Viterbi edit at a random point of a 1 h shoot, clocks uniform in ±40 ppm, a Theil–Sen fit over 12 windows of 2 s, 12 seeds per clip. The synthetic result (no damaged cuts down to −10 dB) does not carry over: the mean share of cuts jumping over one 30 fps frame is 0.34 at 0 dB and 0.64 at −10 dB, and four clips lose 62–97% of cuts at 0 dB, while applause, roll and romans stay at 0 down to −10 dB. Partial whitening (gamma 0.25 instead of PHAT) cuts the 0 dB mean to 0.014 but only to 0.48 at −10 dB. Limits: effects rather than field audio, simulated delay, noise and clocks (±40 ppm is swept, not sourced), event rate and the one-frame criterion assumed, gamma chosen in-sample on the same clips, no viewer test.

Does the capture-kit packing result survive real, sourced device weights, energies and runtimes?

research/capture-realkit · 8 tests

What does the capture-kit packing result look like with real, sourced devices (Insta360 X4, DJI Osmo Action 4, iPhone 15 Pro Max, Sony a7C II and lenses, Anker packs)? Exact search reaches all 18 shots at 4 kg, greedy stalls at 12; the unsourced mount masses move the exact figure by up to 10 shots.

Abstract

Re-runs the capture-kit search (11,520 kits, 6,912 legal) on a catalogue of real devices whose weights, battery energies and runtimes come from manufacturer and retailer pages read 2026-10-21 (Insta360 X4, DJI Osmo Action 4, iPhone 15 Pro Max, Sony a7C II with three lenses, Peak Design tripod, two Anker banks). With a 4 h recording floor, exact search reaches all 18 band × viewpoint shots at 4 kg (the assumed catalogue needed 5 kg) and a greedy packer stalls at 12, as before: the lens-needs-a-body complementarity survives real figures. Sweeping the figures no source gave (phone recording draw, selfie-pole, mini-tripod and mast masses, within ±30%) shows the ranking is robust (greedy is below exact in 77–99% of draws from 1.5 kg up) but the exact variety at a given budget is not: at 4 kg it falls from 18 to as low as 8, so the mount masses, which no source gave, matter more than any camera. Not sourced: those mass and draw figures, the iPhone recording draw (Apple publishes playback only), cell voltages used to turn mAh into Wh, and which bands each device covers.

Does trimming sync windows by their distance from the fitted clock line, with no tuned threshold, beat the peak-sharpness gate on real audio?

research/capture-residgate · 8 tests

Does a clip-free residual-from-the-clock-line trim beat the PSR gate? At 0 dB a little (1.4% to 0.9% of cuts over a frame); at -10 dB no (46% vs 38% for PSR).

Abstract

Follow-up to capture-syncgate. Drops GCC windows more than 2–20 ms off a Theil-Sen clock line (up to 3 rounds), a threshold in time rather than in peak sharpness, over 10 real LibreOffice gallery clips (4 cameras, 12 windows, ±40 ppm, 12 seeds). At 0 dB SNR with gamma 0.25 the share of cuts jumping over a frame falls from 1.4% to 0.9% for any tolerance in the range, mostly from one clip (untie). At −10 dB it helps little (46.1% vs 47.6% ungated) where the PSR gate gets 37.8%; combined 37.4%. It does not fix the held-out gamma-choice failure from capture-syncgate (9.9%). Why trimming drops so many windows: for space2 66% of windows are period aliases (a 12.1 ms near-tone; errors at multiples of the period, 0% under a control period), wallewal 12% aliases and 14% gross, untie and soft 25–34% gross early-start failures of unknown cause. Limits: ten clips from one gallery, synthetic delay and white noise, assumed clock rates, small grids, assumed one-frame criterion.

In a reverberant room, does PHAT win after all, and can the whitening exponent be chosen per window?

research/capture-reverbsync · 7 tests

In a reverberant room, does PHAT win after all? No: across 10 simulated rooms PHAT is the worst fixed weighting; gamma 0.5 cuts failures from 0.26 to 0.09, and choosing gamma per window by peak-to-sidelobe ratio helps less than just fixing it.

Abstract

Adds simulated reverberation (exponentially decaying Gaussian tails, RT60 0–1.0 s, direct-to-reverberant ratio +5 to −5 dB, Schroeder's statistical model) to capture-realaudio's 10 real clips and tests GCC delay estimators for camera audio sync. Across 20 cells (10 rooms, SNR −5 and +5 dB, 200 windows each), PHAT is the worst fixed weighting in 19 of 20 cells: at RT60 0.6 s, DRR 0 dB, SNR +5 dB it puts 0.26 of 2 s windows more than one 30 fps frame off, against 0.09 for partial whitening with γ = 0.5 and 0.20 for plain correlation. Choosing γ per window by peak-to-sidelobe ratio beats PHAT but not the best fixed γ; an oracle selector would roughly halve the failures again. Limits: statistical rooms rather than measured impulse responses, white noise, sound-effect clips, simulated delay.

Does a soft confidence weight beat a hard switch for repairing stereo-depth view synthesis, and does matching cost help?

research/capture-softgate · 7 tests

Does a soft confidence weight beat a hard switch for repairing stereo-depth view synthesis, and does the matcher's cost help? A ramped weight on warp disagreement beats the hard gate by 0.1 to 0.5 dB in every condition (29.5 vs 29.3 dB held out at 15°) and leaves a quarter fewer seams, closing only a tenth of the oracle gap; matching cost is a worse signal than disagreement (level with hole filling, worse with a bystander). Toy scene.

Abstract

A ramped weight on how much the nearest raw camera replaces a stereo-depth synthesis, driven by disagreement between the two warps, beats the hard switch in every condition tried by 0.1–0.5 dB (29.5 against 29.3 dB at a 15° gap on held-out seeds) and leaves about a quarter fewer false edges, but closes only about a tenth of the 2.6 dB gap to a per-pixel oracle. The stereo matching cost is a worse confidence signal than warp disagreement: gating on it is level with plain hole filling and worse under a bystander or sensor noise. Stylised toy scene, parameters tuned on development seeds.

How much footage does a multi-camera travel kit produce, and what limits carrying and backing it up?

research/capture-storage · 15 tests

How much footage does a multi-camera travel kit produce, and what limits carrying and backing it up? Four cameras make ~860 GB a day; cards weigh ~8 g but the evening copy takes 2.7 h on one reader, so a safe plan needs 6 readers and ~450 g (assumed device figures).

Abstract

Data volume, card and SSD weight, nightly offload time and unbacked footage for a rig of 1 to 8 cameras filming 4 h a day for a week, at assumed 60 to 200 Mbit/s bitrates. Four cameras make about 860 GB a day (6 TB a week), yet their cards weigh about 8 g and a camera's battery is 440 to 980 times heavier per recording hour than its card. The binding limit is the evening: one reader copying one day to one SSD takes 2.7 h, so two SSD copies inside 1 h need 6 readers and about 450 g, half an hour is infeasible, and halving the bitrate halves the weight. Figures in the first version are assumed, not datasheets. Update 2026-10-21: action-camera bitrate (120 Mbit/s), card mass (0.25 g microSD) and card bus ceilings (UHS-I 104, UHS-II 312 MB/s) are now sourced, and SHA-256 hashing was measured at about 1,460 MB/s per core; the conclusion holds (copy time binds, not card weight; a UHS-II reader cuts the plan from 426 g to 351 g; verify while copying). The 360-camera, phone and mirrorless bitrates and SSD figures remain assumed.

How often must free-running cameras be re-synced by audio, and does rejecting wild estimates help?

research/capture-syncbudget · 5 tests

How often must free-running cameras be re-synced by audio? At 40 ppm a sync lasts 1000 s for a 40 ms tolerance; a drift gate on fresh estimates helps only if you re-sync every 300 s. Synthetic audio.

Abstract

Schedule study for audio sync of two free-running cameras on synthetic speech-like audio (capture-audiosync's GCC-PHAT estimator). With a 40 ms tolerance (EBU R37 lead limit, a stand-in for a cut threshold) an offset-only sync lasts 1000 s at 40 ppm, so even perfect estimates every 1800 s leave 44% of a 6 h shoot out of tolerance. A drift gate that refuses estimates a real clock (≤100 ppm) could not produce cuts that share from 35% to 22% at −12 dB SNR and from 11% to 3% at −10 dB when re-syncing every 300 s, and removes all wrongly applied syncs at −12 dB; at 900 s or longer it does nothing, since one skipped window already exceeds the tolerance. A wrong first sync is not fixable by a gate. Not real recordings: audio is synthetic, drift is a straight line at an assumed 40 ppm, and the tolerance was set for lip-sync, not for cuts between cameras.

Does a residual audio-sync error show at the cuts of an automatic multi-camera edit?

research/capture-syncedit · 5 tests

Does a residual audio-sync error show at the cuts of an automatic edit? A sync made once leaves 50% of cuts jumping over a frame; a robust fit leaves none down to -10 dB SNR (2 s windows) and then fails outright, with 11 to 24% of cuts repeating or dropping a loud event.

Abstract

Edit quality cannot show it (a camera is shifted in score and picture together), so this counts continuity at each cut: the jump in world time between the two cameras, and loud events inside it. 4 cameras, a 120 s Viterbi edit from capture-edit at a random point of a 1 h shoot, camera clocks uniform in ±40 ppm, sync errors from the real capture-audiosync GCC-PHAT estimator on synthetic audio (12 seeds). A sync made once and never redone leaves 50% of cuts jumping over one 30 fps frame (mean jump 40 ms); a Theil–Sen fit over 12 windows leaves none from 0 dB down to −10 dB SNR with 2 s windows (−15 dB with 5 s). Below that it flips to 69–98% of cuts jumping 0.3–0.5 s with 11–24% of cuts repeating or dropping a loud event. A threshold, not a slope. Synthetic audio, assumed event rate 0.5/s and one-frame visibility criterion; ppm range swept, not sourced; no real footage and no viewer test.

Does a robust fit or a Kalman filter hold a camera's clock better between audio re-syncs than a least-squares line, once the clock drift comes from a real crystal curve?

research/capture-syncfilter · 8 tests

Does a robust fit or a Kalman filter hold a camera clock better between audio re-syncs? On a clock rate computed from a sourced AT-cut crystal curve the plain offset hold is already fine up to 1800 s and every fit ties; only a Kalman filter tuned to an assumed 30 ppm wobble wins. Synthetic audio.

Abstract

Follow-up to capture-syncslope on synthetic speech-like audio (capture-audiosync's GCC-PHAT, residual searched). Replaces its assumed 10–30 ppm rate wobble with a clock rate computed from the AT-cut crystal temperature curve in the Corning/Tele Quarz data book (a3 = 1.05×10−4 ppm/K3) over assumed camera temperature profiles (a 25 K warm-up, a drifting day) and assumed ±5 arcmin cut-angle errors. That curve moves the relative rate only 15 to 20 ppm over 6 h, so the plain offset hold is already within a 40 ms stand-in tolerance 98–100% of the time at 300 to 1800 s re-syncs; fitting a line matters only at 3600 s (37–43% bad to 9–16%). Theil-Sen and Kalman holds tie with least squares there. They differ only on the assumed 30 ppm wobble: a Kalman filter tuned to fast rate wander (20 ppm per √h) keeps 900 s re-syncs to 0.9% bad where every other fit gives 27% and the offset hold 13%, while at 1800 s every hold fails. Base ppm, temperature profiles, cut-angle errors and wobble are assumed; no real recording.

Does finding the start offset first let short audio windows sync two drifting cameras at lower SNR?

research/capture-syncfine · 5 tests

Does a coarse offset first let short windows sync drifting cameras? A 10 s window finds the offset, then 0.5 s windows hold within a frame from -12 dB SNR; plain 1 s windows fail to -5 dB. Synthetic audio.

Abstract

Coarse-to-fine audio sync on synthetic speech-like audio: one 10 s GCC-PHAT window gives the start offset (0.8 s), then every later window on camera B is cut that much earlier so short windows overlap fully. Over 1 h at 40 ppm, 0.5 s windows hold within one 30 fps frame from −12 dB SNR where plain 0.5 s windows never work; 1 s windows go from failing up to −5 dB to working at −15 dB; with reverb at 0 dB SNR 1 s windows go from 0/20 to 20/20 seeds. At −20 dB all fail. Not real recordings: audio, delay and the 40 ppm are synthetic or assumed; the plain baseline loses much of its overlap by construction.

Does dropping doubtful sync windows, with settings chosen on other clips, keep a multi-camera edit in sync on real audio?

research/capture-syncgate · 5 tests

Does a peak-sharpness gate plus settings chosen on other clips keep the edit in sync? At 0 dB yes (1.9% of cuts over a frame held-out vs 34% for PHAT); at -10 dB only partly (39% vs 64%).

Abstract

Follow-up to capture-realedit. Adds a peak-to-sidelobe (PSR) gate that drops GCC windows whose peak is not distinct before the Theil-Sen fit, and picks gamma and the gate threshold leave-one-clip-out over 10 real LibreOffice gallery clips (4 cameras, 12 windows, ±40 ppm, 12 seeds per clip). At 0 dB SNR, gamma 0.25 with a chosen threshold leaves 1.9% of cuts jumping over a frame on held-out clips (PHAT: 34.2%); at −10 dB 39.1% (PHAT 63.7%, ungated gamma 0.25 47.6%). Choosing gamma on other clips alone fails (9.9%) because the one clip where gamma 0.5 collapses cannot be seen from the other nine. Limits: ten clips from one gallery, synthetic delay and white noise, PSR thresholds from a probe on the same clips, and the one-frame criterion is assumed.

Does holding a fitted clock slope between audio re-syncs let free-running cameras go longer without a refresh?

research/capture-syncslope · 5 tests

Does holding a fitted clock slope between audio re-syncs help? At 40 ppm it cuts out-of-tolerance time from 44% to 3.7% at 1800 s intervals, but an assumed rate wobble makes a strict line gate lock out. Synthetic audio; also corrects capture-syncbudget.

Abstract

Follow-up to capture-syncbudget on synthetic speech-like audio (capture-audiosync's GCC-PHAT, window cut with the current correction so only the residual is searched). With a steady 40 ppm clock error, holding a fitted line instead of the last offset cuts the share of a 6 h shoot beyond 40 ms (EBU R37 lead limit, a stand-in for a cut threshold) from 44% to 3.7% at one re-sync per 1800 s and from 72% to 12% at 3600 s; the remainder is the first interval. With an assumed 10 to 30 ppm rate wobble a strict line gate locks out true estimates (69–88% bad at 300 and 900 s); resetting after two rejections fixes 300 s but at 900 s and 30 ppm the plain offset hold wins (13% vs 27%). Also corrects capture-syncbudget, whose raw-window correlation let the delay outgrow the 2 s window. Wobble amplitudes are swept, not measured; no real recording.

Can a delay prior taken from a camera's own earlier windows give the oracle-prior gain in GCC audio sync?

research/capture-trackprior · 6 tests

Can a delay prior from a camera's own earlier windows replace the oracle? Not yet: 22.5% to 37% wrong on the failing clips against 0% for the oracle, and an early alias becomes an hour-long lock-in.

Abstract

Follow-up to capture-aliaswin, whose ±10 ms prior was centred on the true delay and gave 0% wrong windows. Here the prior is a Theil-Sen line through the camera's own earlier estimates (ten real LibreOffice clips, 2 s windows, 0 dB white noise, 0.8 s offset, ±40 ppm drift, 252 scored windows per clip). It does not reproduce the oracle: with one prior-free bootstrap window it is as bad as no prior on the four failing clips (37.4% vs 37.5% wrong) and 37% of those cameras are wrong in every scored window of the hour, because an early alias is tracked rather than corrected; five bootstrap windows reach 22.5% wrong (14% of cameras lost). Capping the half-width under the clip's period breaks clips that worked when the line has no slope yet (33.8% wrong on six clips that had 0%). Limits: constant simulated drift, one SNR, an untuned fit and half-width rule, ten clips. Added 2026-10-29: gating (re-acquire when the unrestricted GCC peak is 1.5× the in-prior peak) cuts the fraction of lost cameras on the failing clips from 36.8% to 7.6–9.7% and the wrong share to 22.1% with one bootstrap window, leaving the six good clips at 0%; the tone clip space2 is not fixed (60–81% wrong) because its aliases have equal peak heights.

Does telling a learned view-synthesis blend which source is wrong help, or does it need to choose between the sources?

research/capture-whichwrong · 11 tests

Does telling the blend model which of two sources is wrong help, or does it need to choose between them? New occlusion and cross-check features add 0.03 dB; a per-pixel source weight adds 0.47 dB (40 of 40 test configs), recovering 0.54 of the headroom to the best blend.

Abstract

Follow-up to capture-learngate. Two ways to close the gap to the best per-pixel blend were tried on the same toy scenes: four new features meant to say which warped source is wrong (signed depth order, and a cross-check that warps each source into the other's camera), and a second network that predicts a per-pixel mix of the two sources before the raw-camera blend. On 40 held-out scenes (new ground textures) the new features move the learned blend from 26.78 to 26.81 dB (+0.03, ahead in 35 of 40); the source weight moves it to 27.28 dB (+0.50 over learngate, ahead in 40 of 40), recovering 0.54 of the 2.71 dB hard-gate-to-best-blend headroom against 0.36 before. The gain comes from the second output, not the new features: with the old 12 features the source weight gets 27.24 dB. Outside the training ranges the gain shrinks to +0.04 dB over learngate. Toy analytic scene, block matcher, PSNR only; every scene parameter is drawn from stated ranges that are assumed, not sourced.

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.