Raspberry Pi has been displaying rising curiosity in on-device synthetic intelligence of late, partnering with Hailo to provide the Raspberry Pi AI Package bundle for the Raspberry Pi 5 — and now with Sony for the Raspberry Pi AI Digicam, an RGB digital camera module with a distinction: it is also an accelerator for pc imaginative and prescient fashions.
Constructed round a Sony IMX500 “Clever Imaginative and prescient” sensor, the Raspberry Pi AI Digicam Module can deal with a spread of mannequin sorts that run on-sensor utilizing an built-in accelerator and 8MB of devoted reminiscence — sufficient for real-time object detection, segmentation, pose estimation, and extra, and all whereas maintaining the Raspberry Pi’s CPU and GPU cores free for different work.
Does the Raspberry Pi AI Digicam Module ship? Let’s discover out.
{Hardware}
- Sensor: Sony IMX500 Clever Imaginative and prescient Sensor (12.3 megapixels, 7.857mm, 1.55μm pixel dimension, rolling shutter)
- Decision: 4,056×3,040
- Framerate: 10 frames per second full-resolution, 30fps 2×2 binned
- Focus: Guide 20cm-∞, 4.74mm focal size
- Area of View: 66° horizontal, 52.3° vertical (±3°)
- F-stop: f/1.79
- Output: Bayer, RAW10, YUV, RGB, Areas of Curiosity (ROI), metadata
- Most Tensor Decision: 640×640
- Precision: INT8
- Reminiscence: 8,388,480 bytes (~8MB) shared between firmware, weights, working RAM
- Measurement: 25×24×11.9mm (round 0.98×0.94×0.47″)
- Worth: $70
First teased at Embedded World earlier this 12 months, the Raspberry Pi Digicam Module options Sony’s IMX500 picture sensor — a 12.3 megapixel shade sensor with 1.55μm sq. pixels. There is a manual-focus lens already fitted, with no assist for interchangeable lenses, and the sensor board mates with the digital camera module board correct the place an on-board Raspberry Pi RP2040 microcontroller handles interfacing the machine studying accelerator with the host Raspberry Pi.
The Raspberry Pi AI Digicam Module is each a 12.3 megapixel digital camera and a fully-functional machine studying accelerator in a single. (📷: Gareth Halfacree)
Whereas that is Raspberry Pi’s first digital camera module to function the IMX500, it isn’t Sony’s first IMX500 product to focus on the Raspberry Pi: the corporate’s official standalone analysis kits for the IMX500 all include a conversion board to attach them to a Raspberry Pi because the driving system. It’s, nonetheless, the smallest: the Raspberry Pi AI Digicam Module has the identical footprint and mounting factors because the Raspberry Pi Digicam Module 3, which means it is drop-in appropriate with many present mounts and instances — although it is barely deeper, so fully-enclosed instances designed for the Digicam Module 3 might not match the AI Digicam Module.
The module connects to the host Raspberry Pi over its MIPI Digicam Serial Interface (CSI) port, as with earlier non-AI digital camera modules, and a flat versatile circuit (FFC) is included. This, nonetheless, is just appropriate for the newest Raspberry Pi 5 and the Raspberry Pi Zero vary, narrowing because it does at one finish; anybody with a Raspberry Pi 4 Mannequin B or different variant with a full-size CSI connector will want a distinct cable.
The module is appropriate with any Raspberry Pi mannequin, bar the CSI-lacking Raspberry PI 400, although some fashions want a distinct FFC. (📷: Gareth Halfacree)
There aren’t any actual shock on the module itself, though using a manual-focus lens is among the few disappointments: the Raspberry Pi Digicam Module 3 was the primary official module to function motorized autofocus capabilities, and it is a disgrace that hasn’t carried throughout to the brand new mannequin. Likewise, the IMX500’s rolling shutter means it is prone to battle with extraordinarily quick movement — however that is hardly on Raspberry Pi as, on the time of writing, Sony had but to announce a global-shutter equal to the IMX500.
Clever imaginative and prescient
Putting in the digital camera is as straightforward as every other mannequin: join one finish of the FFC to one of many mixed CSI/DSI ports on the Raspberry Pi 5, or the devoted CSI port on older fashions, and the opposite finish to the digital camera. Make certain each socket flaps are safe, then energy on.
These used to working pc imaginative and prescient duties on a Raspberry Pi will, at this level, expect to spend fairly a while getting every part arrange: putting in the required software program, pulling a mannequin from wherever, organising the digital camera, and eventually connecting the video feed to the mannequin — throughout which period you may see not less than one of many Raspberry Pi’s CPU cores redline and keep there, until you are utilizing an exterior accelerator.
Fashions like MobileNet SSD (pictured) run on-camera in real-time, taking on no CPU time on the Raspberry Pi. (📷: Gareth Halfacree)
The Raspberry Pi AI Digicam Module is not like that. The software program is built-in into the libcamera stack, and getting began is as straightforward as working rpicam-hello together with your selection of two pre-installed fashions: the MobileNet SSD object detection community or the PoseNet pose estimation community. Choose your poison and wait: the mannequin might be transferred to the digital camera module over the CSI connection, in a course of that is dealt with by the on-board RP2040, and loaded into 8MB of on-device RAM.
For our pre-release testing, we needed to set up chosen packages from a personal repository; this should not be required by the point the digital camera module hits retail. You will want the newest model of Raspberry Pi OS, and solely a few fashions are included within the built-in software program stack; extra might be discovered on Raspberry Pi’s IMX500 “mannequin zoo” GitHub repository, performing as a useful reference for working with the digital camera in Python.
A mannequin mannequin
There is a little bit of a delay if you first decide a brand new mannequin, because it’s transferred throughout to the digital camera. As soon as it is on there a preview window will seem precisely as if you have been utilizing rpicam-hello as regular, displaying a reside view from the digital camera — besides this time it contains the output of your chosen mannequin too, as each machine-readable metadata and an elective seen overlay.
Other than producing the overlay based mostly on the streamed metadata, this all occurs on-sensor: open up a second terminal window and fireplace up the useful resource monitor of your selection and you will see the Raspberry Pi AI Digicam Module is taking on no extra CPU time than the non-AI Raspberry Pi Digicam Module 3 — all of which goes into picture processing for the reside preview, with none required to run the mannequin.
New fashions are transferred to the digital camera’s accelerator over CSI, utilizing the module’s on-board RP2040 microcontroller. (📷: Gareth Halfacree)
Anybody who’s experimented with tinyML — working machine studying fashions on microcontrollers and different resource-constrained gadgets — will, at this level, expect to listen to that the body price is measured in seconds-per-frame fairly than frames-per-second. Fortunately, that is not the case: Sony has achieved one thing really exceptional with the IMX500, and it is able to working each the MobileNet SSD and PoseNet fashions in real-time overlaid on a video feed working at a clean 30 frames per second.
Actual-time pc imaginative and prescient with out tying up a CPU core and and not using a separate devoted accelerator is spectacular; that Sony and Raspberry Pi have been capable of squeeze the expertise into the identical footprint because the common Raspberry Pi Digicam Module 3 is nothing wanting unimaginable. It is a minor energy hog, nonetheless: whereas actively working a mannequin and streaming video to the host, the Raspberry Pi AI Digicam Module accounts for about 1.87W of an general 4.12W energy draw on a Raspberry Pi 5 2GB. The accelerator can also be always-active: if a mannequin has been loaded into reminiscence it’s going to at all times run and stream metadata together with the video, although you’ll be able to disable visualization by excluding the matching post-processing file out of your command.
Expensive zoo
Except you have got very primary necessities, you may quickly tire of watching the digital camera overlay a stick-figure in your gyrating physique or determine a pair of scissors sat in your desk. The Raspberry Pi AI Digicam Module, nonetheless, shouldn’t be a two-trick pony: the {hardware}’s launch comes alongside the opening of a “mannequin zoo,” with three core Python examples for picture classification, object detection, and segmentation.
These examples can be utilized with certainly one of a spread of fashions, together with EfficientNET v2, MobileViT XS, ResNet18, SqueezeNet, and YOLOv8n. Some, like YOLOv8, can run on the most 640×640 decision obtainable to fashions on the digital camera; others are restricted to 320×320, 260×260, 256×256, or as little as 224×224. As you’d anticipate, the extra demanding the community the decrease the restrict.
A “mannequin zoo” supplies Python examples for classification, object detection, and segmentation and a spread of ready-to-use fashions. (📷: Raspberry Pi)
The Raspberry Pi AI Digicam, then, is not a magic wand: its 8MB of RAM must be shared between its firmware, the loaded mannequin weights, and dealing reminiscence. Whereas instruments are supplied for adapting your individual initiatives to run on the digital camera, appropriate with each TensorFlow and PyTorch, do not go into issues anticipating to have the ability to run large fashions: the digital camera has no entry to the Raspberry Pi’s RAM, in order that 8MB is all you have got no matter whether or not you are connecting it to a 512MB Raspberry Pi Zero 2 W or an 8GB Raspberry Pi 5.
Conclusion
At $70, the Raspberry Pi AI Digicam Module is almost thrice as costly because the Raspberry Pi Digicam Module 3 — for apparent causes. In the event you’re not occupied with low-power edge AI, it is a simple selection: persist with the Digicam Module 3 and benefit from the far broader advantages of motorized autofocus. This goes, too, for these engaged on battery-powered initiatives, the place the near-2W draw of the AI Digicam Module may show an unwelcome drain in the event you’re primarily occupied with simply plain video streaming or still-image seize.
When you have any curiosity in any respect in on-device pc imaginative and prescient, although, the Raspberry Pi AI Digicam Module comes extremely beneficial. Having the ability to use all of your CPU and GPU cores for different work whereas an object recognition mannequin runs in real-time is an actual boon, and as every part occurs over the CSI connection you have obtained all of the Raspberry Pi’s USB ports, and the Raspberry Pi 5’s PCI Categorical lane, free for different {hardware}.
Anybody with even the slightest curiosity in pc imaginative and prescient and $70 to spend ought to decide up the AI Digicam Module. (📷: Gareth Halfacree)
For these already experimenting with synthetic intelligence on the Raspberry Pi, the brand new digital camera module is a must have addition to your setup. It may be used alongside present accelerators just like the Hailo-8 or the Google Coral Edge TPU, and there is nothing to cease you working a number of fashions on the similar time — one on the digital camera, one on the accelerator, one on the Raspberry Pi itself.
Whereas it may need been good to see a world shutter, the rolling shutter is way from a deal-breaker — and whereas the fashions themselves are restricted to a most enter decision of 640×640, gaining access to a top quality 4,056×3,040 picture sensor if you want it’s nothing to be sniffed at.
The Raspberry Pi AI Digicam Module is now in the stores from the corporate’s official resellers, priced at $70.