Berkeley Bolted 48 Camera Chips Into One Microscope and Hit 25.2 Billion Pixels a Second

Microscope design has always been a three-way bargain. You can have fine resolution, a wide field of view, or a fast frame rate — improve one and you generally pay for it somewhere else. A UC Berkeley–led team has now built an instrument that refuses the bargain, and published it in Nature Photonics.

The numbers are the story. Their computational microscope captures 3-micron resolution across 5 square centimeters at 120 frames per second, a throughput of 25.2 billion pixels per second. "This is really a breakthrough in the field of computational microscopy," said Laura Waller, professor of electrical engineering and computer sciences and the study's principal investigator. "It's the largest space-bandwidth time product of any practical microscope we know of." That quantity — space-bandwidth time product — is the formal name for the trade-off itself, the total amount of spatial and temporal information an optical system can push through per unit time.

The hardware trick is an array of 48 separate camera sensors mounted on a single circuit board about the size of a credit card. Operating together, they behave like one enormous sensor, which is how the pixel count gets so high. The problem is that they are physically disjoint: there are gaps between the sensors, and light landing in those gaps is simply lost, leaving holes in the image that no amount of stitching can fill.

The fix comes from compressed sensing. The team fabricated a custom phase mask — an engineered diffractive optical element, essentially a glass plate that bends light in a designed pattern — and placed it in the optical path. "Light that would normally fall between the sensors is redirected onto them instead," Waller explained, "and using a computational algorithm, we were able to fill in the data gaps and reconstruct the image." The reconstruction is not guesswork; the mask scrambles the scene in a known way, so the algorithm can invert it.

To demonstrate the instrument, the researchers imaged both static and moving samples. On static targets, the global structure of their reconstructions matched conventional low-resolution images while resolving considerably more detail. On dynamic samples, they recorded dozens of freely moving C. elegans nematodes at 120 frames per second for 15 seconds. "We were able to track the freely moving C. elegans, as well as some structures within them," said lead author Kevin C. Zhou, formerly a postdoctoral researcher in Waller's lab and now an assistant professor at the University of Michigan. From the reconstructed video the team tracked individual worms and imaged their rapid pharyngeal pumping — the muscular action the animals use to feed.

One practical detail may matter as much as the optics. A microscope with 48 sensors and a diffractive element would normally demand painstaking manual calibration; the team's computational approach eliminated it. "I think calibration-free capabilities like our method will prove to be one of the key ingredients for scaling up imaging systems in the future," said co-author Chaoying Gu, a Ph.D. student in the Department of Electrical Engineering and Computer Sciences.

The obvious application is watching many live organisms at once, at high enough resolution to see inside them, for as long as the experiment runs — the kind of measurement that has historically required choosing which of those three things to give up.