When somebody asks you a query that you just have no idea the reply to, how do you sometimes reply? Most individuals’s first intuition is to inform them to “Google it”. This, after all, means to seize a digital machine, launch an online browser, and sort a search question into Google, then to scan the outcomes for a solution. However that is 2024! Expertise has superior tremendously since “Google” first grew to become a verb a pair a long time in the past. Moreover, a text-based question will not be at all times one of the best ways to hunt out a solution — particularly if you would like extra details about a close-by bodily object that’s not straightforward to explain.
A crew on the MIT Media Lab has hacked collectively an answer that they imagine might make it simpler to get solutions to your burning questions. They’ve developed a prototype known as WatchThis that’s mounted on the wrist and makes use of laptop imaginative and prescient and huge language fashions in a novel method to collect extra details about one’s environment. With WatchThis, you merely level and search.
The {hardware} elements (📷: Cathy Fang)
The machine consists of a Seeed Studio XIAO ESP32S3 Sense growth board, which is powered by a dual-core ESP32-S3 microcontroller and helps each Wi-Fi and Bluetooth wi-fi communication. That is paired with an OV2640 digicam module and a Seeed Studio Spherical Show for XIAO with a 1.28-inch touchscreen. A LiPo battery powers the system, and it’s hooked up to the wrist through a strap and a 3D-printed enclosure.
To make use of WatchThis, the show display screen flips as much as face the consumer. The digicam is hooked up to the rear aspect of the show such that it will possibly seize a video stream of what the wearer is pointing it at, and that video is proven on the show. Subsequent, the consumer factors their finger at an object of curiosity, then faucets on the display screen with the opposite hand. This causes the machine to seize a picture of the scene.
WatchThis in motion (📷: Cathy Fang)
A companion smartphone app is used to kind a query. That query, together with the captured picture, are despatched to OpenAI’s GPT-4o mannequin through the official API. This mannequin can analyze each pictures and textual content and cause about them to reply questions. When the reply from the mannequin is returned it’s displayed on the display screen, on high of the captured picture, for a number of seconds earlier than the machine returns to its regular working mode. Typical response occasions are within the neighborhood of three seconds, making WatchThis moderately snappy.
The builders selected to make use of a smartphone app to permit customers to kind their questions for accuracy, however having to make use of one other machine to kind out a query is a bit clunky. One query instantly raised by this association is why the whole system doesn’t simply run on the smartphone. It already has an built-in digicam and definitely has the power to make an API name, in spite of everything. A voice recognition algorithm, whereas it could not be as correct, might make WatchThis far more pure and environment friendly to make use of. Maybe after some enhancements like this, we are going to inform folks to “WatchThis it” sooner or later. Hey, “Google it” didn’t at all times roll so simply off the tongue both, you understand!