High-speed Imaging through Turbulence
with Event-based Light Fields

Yu-Hsiang Huang1, Levi Burner1, Sachin Shah1, Ziyuan Qu2, Adithya Pediredla2, Christopher A. Metzler1
1University of Maryland, College Park    2Dartmouth College
arXiv Code (coming soon)

A static checkerboard imaged through strong atmospheric turbulence at 120 fps. Our event-based light field camera (right) reconstructs temporally stable edges compared to single-view reconstruction (left).

Abstract

This work introduces and demonstrates the first system capable of imaging fast-moving extended non-rigid objects through strong atmospheric turbulence at high frame rate. Event cameras are a novel sensing architecture capable of estimating high-speed imagery at thousands of frames per second. However, on their own event cameras are unable to disambiguate scene motion from turbulence. In this work, we overcome this limitation using event-based light field cameras: by simultaneously capturing multiple views of a scene, event-based light field cameras and machine learning-based reconstruction algorithms are able to disambiguate motion-induced dynamics—which produce events that are strongly correlated across views—from turbulence-induced dynamics—which produce events that are weakly correlated across view. Tabletop experiments demonstrate event-based light field can overcome strong turbulence while imaging high-speed objects traveling at up to 16,000 pixels per second.

Results

5.1 Light Fields Improve Temporal Consistency

Experimental static checkerboard (120 fps). Video version of Fig. 6 (top) in paper.

Experimental spinning dot on LCD (120 fps). Video version of Fig. 6 (bottom) in paper.

5.1 Light Field Outperforms Single-View Reconstruction

Simulated Results

Simulated single-view results are at 768×768 resolution, which has the same number of input pixels as the 3×3 light field whose views are simulated at 256×256. This ensures a fair pixel-budget comparison between the two configurations.

Video version of Fig. 7 (left, simulated) in paper.

Experimental Results — Sim-to-Real Transfer

Without any fine-tuning on real data, the same model trained on simulated turbulence generalises directly to our tabletop setup (candle field + space heater, 30 ft range). This sim-to-real transfer works because the cross-view disentanglement learned on synthetic turbulence statistics transfers to real-world anisoplanatic conditions.

Video version of Fig. 7 (right, experimental) in paper.

5.2 Event Light Fields Overcome Motion Blur

Comparison with MambaTM (frame-based SOTA) on simulated high-speed scenes. Motion blur is simulated by aggregating 4 consecutive frames into one blurred frame. Event-based light fields directly operate on the event stream and reconstruct the underlying scene without blur from integration. Frame-based methods (MambaTM) cannot compensate for motion blur when scene dynamics or turbulence are fast.

Video version of Fig. 8 in paper.

5.3 Recovering High-Speed Video Through Turbulence 600 fps

Experimental spinning reflective stripe (600 fps). Video version of Fig. 9 (top) in paper.

Experimental bouncing ball (600 fps). Video version of Fig. 9 (bottom) in paper.

Experimental Water Balloon Bursting 600 fps

Video version of Fig. 10 in paper.

Experimental Nerf Dart at 12,000 fps 12,000 fps

A Nerf dart traveling at 16,000 pixels/second is reconstructed at 12,000 fps.

Video version of Fig. 11 in paper.

Characterization of Tabletop Turbulence Strength

To quantify the turbulence severity in our tabletop setup, we image a regular dot grid through the candle field and space heater and compare it against the undistorted ground-truth grid. The geometric distortions and temporal fluctuations visible in the turbulent sequence demonstrate that our setup produces strong, spatially-varying anisoplanatic turbulence.

Imaged target

Dot grid target

Under turbulence

Without turbulence

Additional Experimental Results

Dynamic Checkerboard600fps

A moving checkerboard imaged through turbulence. Unlike the static teaser, both scene motion and turbulence are present simultaneously, which is the core ambiguity that single-view cameras cannot resolve.

Nerf Dart Shot into Water Tank300fps

Nerf darts are fired into a water tank through the turbulence source. The darts enter and bounce inside the tank, producing fast, complex dynamics.