The idea of installing a software update on your fridge already feels kind of weird, let alone one centered around improving its AI capabilities. But that’s exactly what’s happening to Samsung’s line of Bespoke refrigerators this week, and to my surprise this patch is making major strides at providing truly useful machine learning in a modern day icebox.
As a quick recap, Samsung has offered AI-powered features like automatic food recognition and meal planning on its Bespoke refrigerators for a couple years already. However, as I found out after reviewing its flagship model late last year, the company’s AI capabilities are still very much a work in progress. Previously, the fridge could recognize around 60 different kinds of fresh foods (like fruits and veggies) alongside another 50 or so packaged goods like yogurt or popcorn. That felt like a decent start, but considering the sheer number of different items you can find at a typical grocery store, it was far from complete. Furthermore, you often had to input additional data like the number of items or when something was first added, which made the idea of AI-assisted grocery tracking more tedious than I’d like. I don’t know about you, but I generally don’t want to have to type on my fridge and I’m pretty sure the engineers at Samsung agree, which is probably what brought about this major update to its Bespoke refrigerator software.
So what’s new?
The big change is that Samsung is adding support for Google Gemini, which has several important implications. By combining Samsung’s existing on-device object recognition with Google’s cloud-based models, the total number of identifiable foods is increasing from just over 100 items to more than 2,000. Now this does mean you will need to connect the fridge to Wi-Fi, but considering the number of other smart features it supports like calendar integration and video playback, that’s not a big ask.
Another update is that Samsung is using Gemini to expand voice controls, allowing users to ask the fridge to change things like device settings, check details like when the water filter was last replaced or to even help troubleshoot issues. And depending on the situation, the fridge can even play back a tutorial about how to solve the issue.
Alternatively, for more complicated or harder-to-solve problems, Samsung is introducing what it calls Reliability AI, which is designed to monitor the fridge’s components or help identify faults before they get too serious. Or in cases where the device needs to be serviced, the AI can provide more detailed info to agents while potentially allowing them to fix certain things remotely.
For example, a Samsung representative told me that if a customer calls and says that cubes from the icemaker are coming out in clumps and stuck together, Reliability AI could allow agents to reduce the amount of water that is being added to the ice tray — all without ever needing to physically come to your home. Critically, Samsung says that while the fridge will monitor and track device health metrics, owners will need to provide express consent in order for repair personnel to access that data. Meanwhile, if an issue does require in-person servicing, Samsung says that by sharing this data with repair technicians, it allows people to identify and solve problems faster instead of having to arrive with no context and diagnose issues from scratch.
How this works in the real world
My test unit hasn’t run into any mechanical issues in the eight months I’ve been using it, so I haven’t been able to evaluate Samsung’s Repairability AI. That said, after checking it out first at Samsung’s headquarters, I’ve had the chance to use an early version of the Bespoke line’s new software over the last two weeks — including its upgraded support for cloud-based object recognition — and the improvement is profound. Even after using it for a while, I’m still surprised by how many different foods it recognizes. Deep in the back of my fridge, I have a can of Bull Head Shallot Sauce, which is a rather niche ingredient from Taiwan used almost exclusively in Asian dishes. However, the AI had no trouble recognizing it, automatically tagging it and including when it was first added to the fridge’s AI Food Manager.
On top of that, the system is now much better at recognizing brands and counting the number of specific ingredients in order to create more detailed listings. It can distinguish between a Diet Coke and Coke Zero while also accurately noting that there were multiples of each item. And even though the fridge often has to ping a cloud-based server somewhere to help recognize various items, results appeared rather quickly, often in less than a few seconds. And for certain foods like avocados, I was delighted that the fridge tracks how long you’ve had it and will surface a notification that it might be getting close to expiration. Granted, it’s not always right, but all I really need is a reminder to check on things and it does just that.
Sam Rutherford for Engadget
I also noticed that the fridge now remembers when you frequently take a specific food out and then asks if you want to add that item to your shopping list. It’s a nice reminder to replenish staples you use regularly and happens in a low-friction way, so it doesn’t become annoying. From there, you can simply check your phone when you’re at the store instead of needing to manually curate a list every week. Also, because the fridge does a much better job of recognizing and tracking what’s inside, it can provide better suggestions about recipes you can cook using ingredients you already have.
That said, like a lot of current models, the AI doesn’t always nail every detail. For example, I was initially impressed when it automatically labeled a tub of fake cream cheese as “Philadelphia Plant-based,” until I realized that the label was incomplete and the AI was merely reading what was written on the lid and didn’t have the smarts to accurately finish the description. Don’t get me wrong, it provides more than enough info to help me figure out what’s in the fridge when I’m glancing at the Food Manager. It’s just not quite spot on.
Outlook and things that still need work
The one issue with this update is that like a lot of AI services today, Samsung’s new software can be a bit overconfident or prone to hallucinations. One time, as my wife was putting something back in the fridge, the algorithm took a picture of a brightly colored bandage on her finger and labeled that as a veggie, which it very much is not. Other times it seemingly just guesses. But I’d argue going from around 100 identifiable items to over 2,000 is a very welcome improvement even with the limitations.
The other weird thing is that even though Samsung is leveraging Google’s AI models for a lot of the fridge’s new features, you won’t see any obvious callouts to Gemini inside the device itself. That’s kind of a bummer because Bixby is still the only digital assistant you can use and talk to directly.
I’ve said before that Samsung’s AI food recognition is a work in progress and I think that still holds true. With this latest update, the company has gotten a lot closer to delivering on the promise of a fridge with truly useful AI-powered features. What once felt more like a promising tech demo has quickly become a handy tool to keep track of your groceries, even with some hiccups here and there.