Microsoft Copilot Plus hands-on: Does it need a Recall?

The biggest topic at Computex 2024 in Taipei is undoubtedly AI-enabled PCs, specifically Copilot Plus PCs. Laptops with this designation receive a host of extra Microsoft AI features, most notably Recall, Image Cocreator, and Live Caption functionality. We went hands-on with these features at Qualcomm and OEM hands-on sessions to better understand how they work.

It’s worth stressing that these laptops aren’t quite out yet — the first wave of devices will be released on June 18. So, while these features might still receive some last-minute polish, what we saw won’t be far off what the consumer experience will look like.

Recall: Search for everything you ever did

Android Authority‘s C Scott Brown already went hands-on with Recall (see the Short above) — probably the most hyped Copilot Plus feature out of all its functions.

Recall is an on-device tool that lets you search for any past activity on your PC, displaying text and visual matches along with a timeline of activities. From these matches, you can copy images and text or launch the app in question. To be blunt, I think it’s very hit-and-miss at the moment.

My first disappointing example was when I visited Reddit in the Edge browser and then popped back into Recall to search for it. The result? No accurate visual or text matches. I also tried visiting the Android Authority website (what a shill) on Edge and then searched for “smartphone news website,” and it didn’t appear as a visual match.

Filtering results is also incredibly inconsistent. Why can I filter by Paint, Edge, and other factors for one search, but filtering disappears for other searches? I’m guessing this is a bug or possibly a case of searches somehow determining which filters get used (if any), as the filters appear on a second outing with the feature.

A different annoyance I discovered is that if you disable the “save snapshots” section in Recall’s settings and then later re-enable it, the Recall functionality itself doesn’t seem to be fully enabled. So you have to re-enable saving screenshots and then head into the system tray to fully re-enable Recall from there. It took me 20 minutes after re-enabling “save snapshots” to realize that Recall wasn’t actually working properly even though I could open it.

It’s not all disappointing, though, Recall can sometimes be great. For example, I conducted a Google search for Shiba Inu dogs in Edge (RIP Kabosu) and then looked for it in Recall. I was pleasantly surprised to see that it found an image of the dog breed on a website that previous users had visited. What was equally as impressive was that the dog was such a tiny part of the screenshot that I actually missed it the first time I looked at it. This moment certainly made me realize that Recall has potential. It also served up accurate visual matches in many other situations, including finding “whiteboard” and “teacher” after watching a YouTube video featuring a language tutor in front of a whiteboard.

One area that’s ripe for improvement is resuming activities. You might think that you can resume any activity after tapping on a text/visual match and launching the app via this page, but that’s not the case. Sure, images can be viewed in the Photos app again, web URLs can be seamlessly launched, and some other tasks will work just fine. But you can’t resume a picture-editing session in the Recall timeline from yonks ago if you haven’t manually saved it, for one. These kinds of limitations are understandable, as I’d imagine that this would require a ton of system resources, but it’s nevertheless something worth noting.

Perhaps most troubling are the serious privacy concerns about Recall, as per security researcher Kevin Beaumont. For a start, sensitive Recall data is apparently stored in a plaintext database. Plus, Microsoft says Recall captures passwords, too, if the password field on a website doesn’t blank them out. Sure, you can exclude some apps and websites from Recall, but Microsoft needs to address these aforementioned issues, especially given they’re enabled by default.

Live Caption and Image Cocreator

Microsoft’s take on Live Caption functions in pretty much the same way as Live Caption on your phone. It’ll give you captions in English if you prefer to read rather than listen, but also translates a claimed 40+ languages into English. By contrast, Live Caption on Android/Pixel translates from or to Chinese, English, French, Hindi, German, Japanese, and Spanish. Microsoft’s captions can be moved around for your convenience too.

There is an understandable delay compared to baked-in captions and subtitles. However, it seems on par with Live Caption for Android at worst and a little faster than Google’s effort at best. Score one for accessibility.

Finally, I got Image Cocreator, which you can find in the Paint app. You simply draw a picture and then use the Cocreator prompt field to spruce it up or reimagine it using generative AI. This all happens on-device too, thanks to beefy AI silicon. For example, you can draw a picture of a house and sun and ask Cocreator to turn it into a better drawing.

I think Cocreator works best when you put a little effort into your drawings or are more specific with your prompt (specifying what exactly it is that you’re drawing). There’s also a creativity slider that controls how imaginative the AI gets. However, the slider is a little arbitrary about the point at which your picture is reimagined (is it zero? Nope. Is it 60? Nope. But it’s 100). It’s also prone to some typical AI weirdness when it comes to humans, although my drawing skills aren’t a great base to begin with.

Are Copilot Plus features worth the hype?

These features will be available on Snapdragon PCs first, with AMD and Intel following later this year. So is it worth picking up a Copilot Plus PC purely for these features?

It’s still too early to say for sure, but my experience felt more like a test than a polished product. Live Caption is a long-overdue, great addition for accessibility purposes. I can’t see myself ever using Paint’s Cocreator functionality, but I imagine it will be handy for creative types wanting to brainstorm image ideas if they don’t have a web connection (such as on a plane).

Then there’s Recall, which is meant to be the big selling point. We’ll need to spend far more time with it and it will likely improve in the buildup to its June 18 launch and beyond. However, between the privacy concerns, the hit-and-miss results, and other areas that need improvement, it still needs more time in the oven.

In fact, I’d argue that the much longer battery life is a better selling point for Snapdragon PCs than most of these Microsoft-backed AI features.