Hands-on with Gemini's 'Ask This Page': Another AI tool that needs a lot of work

At I/O 2024, Google announced a slew of generative AI-powered features across all its major products. In fact, the company made a little joke at the end of the keynote by tallying how many times someone on stage said the term “AI” (it was over 120 times). However, as is Google’s wont, not many of the features the company launched that day are actually available for the public to use yet. Today, though, Android Authority got an early look at a Gemini feature launched at I/O called “Ask This Page.”

As its name suggests, Ask This Page allows you to glean specific information from a webpage by first having Gemini “read” it. Think of it like an interactive personal assistant. It does the heavy lifting for you by reading the whole webpage, and then you can simply ask it for the specific information you’re looking for, thus saving you a ton of time.

This is very similar to two other “Ask This…” features Google launched at I/O: “Ask This PDF” and “Ask This Video.” We already had the opportunity to test out Ask This PDF at I/O on a loaner Pixel, and it worked really well (check the video embedded above for that experience). However, that test was in a very controlled environment on a device that wasn’t ours, and it had only one test PDF. With our early access to Ask This Page, though, we had a lot more time to push the system to see if it has any cracks — and sure enough, it does.

Before we dive in, let me be upfront and say this is all based on an early look at this feature. It is possible Google could make significant changes before it rolls out to the general public. In other words, take everything here with a grain of salt.

Gemini’s Ask This Page: How it works

To activate Ask This Page, you simply pull up the Gemini overlay while looking at a webpage on your Android phone by holding down the power button. Since Gemini is now context-aware — a topic Google spent a significant amount of time discussing at I/O 2024 — it will know that you’ve pulled it up over a webpage. This will trigger the Ask This Page icon you see in the image above.

Tapping that prompts Gemini to scan the page. This can take a little time, the amount of which will depend on how long/complex the page is. Once it’s ready, it will give you a text box prompt saying, “Get help with what’s on this page.”

Once you see that prompt, you can ask questions about the page in natural language. Check out some screenshots below to see how this worked on an Android Authority article about a new Microsoft OneNote feature leak.

In this example, we pulled up the Gemini overlay over the article, scanned the page, and then asked whether or not the feature described in the article is actually released. You can see Gemini’s answer in the third screenshot: “According to the article, the reminder feature is currently under development and not yet released.” This is accurate, so we have a great test run for Ask This Page!

However, this was a very simple test. Let’s find out what happens when we push the limits.

Ask This Page is siloed to one page…but only sometimes

My first gut reaction for how to test Ask This Page was to feed it false information and see if it could use its broader understanding of the web to give proper context to any questions I ask it. In other words, if I’m reading a website that has false information and ask Gemini about that article, will Gemini just feed me back the false info?

In this test, that’s exactly what it does. We pulled up a silly satirical (read: completely made-up) article from The Onion and asked Gemini questions about it. In the article, a mother named Dina Marchesi (who does not exist) refuses to believe that she repeatedly makes snide comments to her daughter and others. For this article, we asked Gemini, “Did she tell her daughter about her dress?” Gemini confirms that Dina Marchesi did comment on her daughter’s dress. Check out the screenshot below for the full response:

Let’s break down this response a bit. First off, Gemini dutifully answers the question without giving any context to the fact that this webpage is The Onion — arguably the most famous satirical news site ever. It does have a disclaimer at the bottom about how Gemini “may display inaccurate info, including about people,” which seems to be a catch-all way to account for situations like this. Still, this is The Onion, and you’d think Gemini would point that out. What if this was a site with false info that isn’t as open about it as The Onion is?

The response also clearly lists the source of the information Gemini provides, which, in this case, is this solitary article. This means we’re not getting any broader context from the internet, at least not for this question.

Essentially, this example makes it seem like Ask This Page is a more complex version of an AI summary. Instead of summarizing an article into a few core bullet points, it lets the user trigger summaries about specific information contained therein.

At this point, we thought this was the limit of Ask This Page. But then we tried a few more experiments and found that it’s more open-ended than it would seem.

Ask This Page? More like ‘Ask Parts Of This Page’

There’s a lot more to a webpage than just text. For this next set of tests, we wanted to see if Gemini could glean information from the entirety of a page, including images, tables, comments, etc. To do this, we fed Gemini our recent hands-on article with the new Chipolo trackers.

Let’s start with images. We asked Gemini if there’s an acceptable use policy for the new Chipolo trackers. The answer is “yes,” which a human reading the article would see by looking at one of the screenshots on the page. Unfortunately, Gemini didn’t find this info in that image. It did, however, find the information on the web:

This is a very confusing outcome. The first sentence of Gemini’s response is, “The document linked doesn’t explicitly mention an acceptable use policy,” except it very much does, just in an image instead of in the text. For whatever reason, Gemini couldn’t find that information, so it went hunting elsewhere. This proves two things: Gemini can confidently tell you that what you’re looking for isn’t on a page even though it is, and that Ask This Page can seek broader context from the internet for certain questions. In other words, the previous issue with Gemini not acknowledging that The Onion article is satire isn’t due to an inability to access information outside of the specific webpage you’ve fed it. Google just doesn’t have a safeguard in place for those situations.

Next, we asked Gemini if the new Chipolo tracker is water-resistant. This information isn’t mentioned in the written text of the article, but it is shown to have an IPX5 rating in an included specs table. This time, Gemini had no issue finding that info:

OK, so images don’t work, but specs tables do. What about comments? In the article itself, there’s no mention of the tracker supporting a left-behind notification. However, this information is discussed by the article’s author in the comments. Let’s see what happens when we ask Gemini for this information:

Once again, we have a situation where the information is on the webpage, but Gemini doesn’t find it and then confidently tells us it’s not there before searching the web instead.

Finally, can Gemini tell us more about who actually created a webpage (or, in this example, who wrote the article)? This article’s author, Rita El Khoury, is a staple of the tech journalism world, with literally thousands of articles written over a career that’s lasted for nearly two decades. Here’s what Gemini had to say when we asked for more information on the article’s author:

In this case, Gemini simply didn’t answer the question. It had plenty of information on Rita on the page itself (if you tap Rita’s photo in the byline, you’ll find a ton of info about her), and it did not search the internet for further details, of which there are plenty.

Gemini Ask This Page hands-on: Can you trust it?

All in all, this feature is not terrible. It certainly does not tell us to put glue on pizza, drink urine, or eat rocks, as Google’s AI Overview results have recently done. That’s a low bar, but Ask This Page rises above.

Still, there’s so much lacking here. There is no warning that the page you are reading is blatant satire. It seems to pick and choose when it wants to search the web for further info/context or just stay on the one page you’ve fed it. It can’t “read” an entire webpage, with things like images and comments hiding valuable details that you’d likely want to know about. And, sometimes, it just doesn’t answer the question even though it has the ability to do so.

As I mentioned at the top of this article, this feature hasn’t officially rolled out yet, so there’s still time for Google to make some tweaks. However, based on Google’s push for AI features to go live, regardless of whether or not they are ready, it wouldn’t surprise me if Ask This Page launches in its current state with few alterations.

What do you think? Based on our experiences here, will you use Ask This Page when it goes live? Let us know in the poll above, and be sure to hit the comments to explain your answer.