Sony's AI image sensor bakes camera smarts right into the chip

Sony's camera sensors are already inside many of the high-end smartphones, but its new AI image sensor promises to make things even smarter – and high-end features cheaper, too. The Sony IMX500 not only includes a 12.3-megapixel CMOS capable of 4K resolution video, but a dedicated AI processor that can do machine learning and more.

Typically, the image sensor is just that: a sensor that captures images. What actually happened to those stills and videos is down to the rest of the device, and we've seen several phone-makers develop their own dedicated co-processors focused solely on crunching their way through shots, applying AI-based HDR and other effects, and using machine learning to identify scenes and people.

The Sony IMX500 promises to do all that, but on the single sensor chip itself. The pixel chip is stacked on top of a logic chip, the latter not only doing the conventional image sensor operations, but having an ISP for image signal processing, and an original DSP dedicated to AI signal processing. It also has memory there, too, exclusively for the AI model to use.

Benchmarks for artificial intelligence tasks are something of a flexible thing right now, and comparing machine learning performance can be tricky. Sony says the IMX500 can do 3.1 milliseconds processing for MobileNet V1, for example, which does object recognition. The whole thing could be completed within the space of a single video frame.

Obviously, this isn't the first time we've seen AI used in photography and video. Sony's argument for the new sensor, though, is that it has the benefit of affordability, security, and performance all in one. Cloud-based processing is relatively cheap, but you have to get the content to and from the cloud, and there are privacy concerns. On-device processing with a dedicated co-processor, as we've seen Apple, Google, Huawei, and others try, is faster and more secure, but it adds cost.

Sony's solution promises both performance and privacy. It'll also be flexible, since device-makers will be able to cook up their own AI model and load it into the embedded memory.

The IMX500 isn't expected to show up in smartphones, at least not initially. Sony's intended audience at first is industrial and task-specific cameras, for example in retail locations or factories. An IMX500-based camera loaded with a people-tracking AI, for example, could spot how many people are passing through a store, and analyze which areas are most popular. In a factory, it could track product and highlight issues in production or staffing.

For connected security cameras, meanwhile, the sensor could be loaded with an AI trained to differentiate between animals and pedestrians, or to highlight when a delivery truck is pulling up versus just driving past. That might make for more useful notifications, as well as cut out cloud processing costs.

The camera itself can shoot 4K video at up to 60 fps, or 1080p video at up to 240fps. It dips to 30fps if you want full AI processing at the same time. For stills, it'll shoot up to 4056 x 3040 images.

Sony says it started shipping samples of the bare chip last month, and expects to ship the packaged product from June. It's likely to catch the attention of the Sony AI team, too, the company's recently-formed division that aims to apply sensor tech to artificial intelligence, robotics, and photography. The first products to actually hit the market using the IMX500 are likely to come in Q1 2021.