New Eyes for Computer Vision

SEBASTIAN BAUER: I'm Sebastian Bauer, CEO and co-founder of Ubicept, and we are commercializing single photon technology to provide on the one hand, the best imaging possible, the best camera images, and also we can see around corners.

TRISTAN SWEDISH: My name is Tristan. I'm CTO and co-founder at Ubicept. I graduated from MIT last year with my PhD and I did my research work with Professor Ramesh Raskar. And so Ramesh has worked a lot on taking some very interesting measurements of light and doing magic with it.

And so about a little over 10 years ago, he did some work with a postdoc at the time named Andreas. And they developed a system that could use light and a system that detected that light that was used for chemistry experiments. But instead of chemistry experiments, they took that data and they turned it into a camera. And the thing that was special about that camera is that it could see light at a trillion frames per second.

And so when you can see light at that time resolution, what they showed is that you can take that data and use it to see around corners. And it's kind of similar to how radar works or sonar. You send a pulse of light out. You see how it scatters through the scene. And then by measuring how long it takes for light to come back, you can use that information to reconstruct a view around the corner.

And so that was like, that was incredible. But it required this weird instrument that was in a chemistry lab at MIT. And it really said that the data was there but we didn't know how to practically detect it. Since then, Andreas went on to become a professor at University of Wisconsin Madison, and started using these other detectors that were quickly dropping in price. They were being used in smartphones today.

And they had all the qualities to be able to timestamp these photons that are necessary to see around corners. Sebastian joined as a postdoc in Andreas' group. And I did my PhD with Ramesh. And what we realized as we were working on these problems is that these sensors are becoming to the right price point, our algorithms were getting good enough, and we could take that original vision 10 years ago in Ramesh's lab at the MIT Media Lab and realize that in a practical product today.

SEBASTIAN BAUER: So when you capture light, fundamentally, that light is composed out of individual small particles called photons. And in an analogy, you can think of them as raindrops. And what conventional sensors are, they are an array of very small buckets.

And these raindrops fall into these buckets and over time a certain volume of water accumulates in each individual bucket, and that is being read out by the electronics. And that means when it's super-bright, a lot of water volume, many photons or raindrops fall into that bucket. And when it's not very bright then very few photons accumulate.

And that works well, obviously, in conventional daylight conditions like these. But the problem is when it's super-bright, you have so many photons falling into each individual bucket that the bucket just overflows. And then your sensor is completely saturated. All you're seeing is white. You don't see any details anymore, any contrast.

That's a problem. If you don't accumulate enough light it's just not possible to read that out or get any data out of that. And the third problem is fast motion. So that means when you have this array of buckets, you have an object that moves here and you have a raindrop fall into this bucket, into this bucket, and this bucket. And that means that this object blurs out in the image.

And these are fundamental shortcomings, so low light, high speed, and also brightness variation, super-bright, super-dark in the same image. However, when you time tag each individual raindrop, assign it a time interval that can go below 1 nanosecond, which is below 1 billionth of a second, and this process is repeated extremely fast, you have all the information available, tons of photon detections. And then you can do processing on top of that and correct for motion, for example.

You can play all kinds of tricks to come up with a truly software-defined single-photon camera that provides the best image capability possible. This single-photon imaging technology is found on the latest iPhone 12, 13, and 14 Pro models, and there it is part of the LiDAR system. So the sensor, the LiDAR system has two sensor halves. One is the laser array that sends light into the scene, and the detector half waits for the light or the individual photons to come back and measures the travel time with a very high time resolution.

And that's an active system. We can kind of redo that to see around corners as well. But most importantly, that detector technology, which is available at this point only at relatively low resolutions, when you make high resolution arrays, these are fundamentally the best cameras, because as I mentioned before, you can capture each individual photon that is around us with a very high time resolution.

And you have the best information that physics allows you to get. And this high time resolution allows you to circumvent these problems that conventional cameras have. So imaging in high speed environments, especially low light, and also when there is huge brightness variations.

TRISTAN SWEDISH: These single-photon detectors have been developing for quite some time. And the first commercial application of these was for distance sensing in cell phones. So they're a single pixel, and these are sensors that there's billions of them in almost every cell phone that you buy today. So single pixel has been fully commodified in the sense that it's low cost, easy to use, and it's sensitive to light.

And the recent development, in I'd say the last decade or two, is going from a single pixel to many pixels. And it turns out that there are some challenges that need to be overcome. But those have been solved in the last 10 years. So now we have arrays of these pixels that look like conventional image sensors.

So you can now go from a single pixel to an array like you have in your camera. And since those challenges have been overcome, they can be produced at scale, meaning that the cost curve matches that of conventional image sensors. They're produced by largely the same technology at foundries that produce image sensors today.

And that means that fundamentally the ability to produce low-cost sensors is on the cusp. I mean, it's happening. The challenge is, how do you take the data that's produced by these sensors and do something useful with it.

And so we bring that capability, as well as these rapidly commodifying sensors together, in order to realize the potential of single-photon imaging. And so really where I think at this huge shift in the way imaging works, it's sort of an overnight change that's taken decades to develop.

SEBASTIAN BAUER: And I should add that we don't make these sensors, obviously. There's big companies working on them. But we ride that wave and have a know-how to repurpose them as general image sensors.

TRISTAN SWEDISH: We formed Ubicept. We saw these sensors were developing at a rapid pace. And we understood the quality of data that could come from capturing light with this time resolution. What we realized is that you can also turn the laser off, so the light source off. So previously we had a system where you had a source of light. You flashed the light into the scene and you look at those echoes of light return.

But if you turn that system off, there's also just light all around us, of course. I mean, we can see with light that is coming from the sun or artificial lights. And what we realized is by time-stamping those photons, not necessarily ones that we introduced in the scene, we can also do conventional imaging in a much different way. And so as a company, we realized that we can really build expertise and the ability to take advantage of single-photon imaging as a whole, both for conventional imaging that could have applications for vehicles or moving platforms, autonomy, security, and then also we could put that laser back on that system and use those systems to also see around corners.

So that's been the journey that we've gone on once we've gone from the lab is how do we take this set of information and capabilities and turn it into practical products? As a company Ubicept's gotten to the stage where we are de-risking a lot of the technology. So the fundamental technology is being de-risked. And we understand what we can and can't do in the short term.

And so we're creating developer kits, evaluation kits for other partners to have this technology in their hands, and they can see what it can do. So we are shipping cameras to partners that can take this data and understand really the power of this new way of sensing. And this is really on a path to scale.

So we're creating an example reference hardware that is using off-the-shelf components. And we couple that with our software. And combining these two things is a reference design that can be incorporated into a product with a partner.

So we're looking for partners to develop this technology alongside with. We're not selling a camera that the average consumer could buy. We're incorporating this technology into a larger system. And so we're at the stage where we're packaging that and integrating that with partners.

SEBASTIAN BAUER: So what we have achieved so far is we have de-risked the technology quite a bit, which is great. We know now what we can do. It's running in real time. It's running on small hardware. And in the next one or two years, the idea is to make it smaller in its scale and package that into much smaller form factor and scale, get it on as many platforms as possible.

And we are super-excited to kind of lead this paradigm change in imaging. And I'm looking forward to working with many more partners and other stakeholders that will join us in creating and riding that wave at the same time.

[MUSIC PLAYING]