Every Google Gemini AI-powered feature for Pixel phones so far

At Google I/O in May 2023, the company debuted its multimodal AI known as Google Gemini. During that launch, Google talked about what Gemini can do and how it will transform many of the company’s products. And since then, the company has only made bolder promises that have only just started to become reality.

Now that Gemini is out there in the world on the Pixel 8 and Pixel 9 series, we’ve made a list of all the things it can do on a Pixel phone so far, and a few brief ideas for features we hope to see in the future.

Which version of Google Gemini do Pixel phones use?

Gemini is Google’s next-generation AI architecture. It’s what’s known as a large language model (LLM). It has been designed from the ground up to replace Google’s former AI model known as PaLM2. For years, PaLM2 has worked under the hood for various Google AI features and products, but now Gemini is the focus. Think of Gemini as being similar to GPT-4, the underlying LLM architecture that powers the popular ChatGPT.

The big difference with Gemini when compared to PaLM2 or even GPT-4 is how it was built. Most multimodal models are built piecemeal — you build one thing to do Task A and then another thing to do Task B and then put them together so the system can achieve Task C. However, Gemini, according to Google, is “natively multimodal, pre-trained from the start on different modalities.” This should, in theory, make it faster, more accurate, and more efficient when it comes to performing its tasks. It should also allow it to perform tasks that other systems cannot.

There are three versions of the Gemini LLM, with each designed for specific types of work. Gemini Ultra is the BIG one. This is the multimodal model that will do incredibly complex tasks. Gemini Pro will power Google’s consumer-level products that work in the cloud. And finally, there’s Gemini Nano, which is designed to work on a device natively and is what appears on Pixel phones. This is the least powerful version of Gemini, but it could be the most practical to use since it will not require a data connection and can be incorporated directly into consumer-level products.

Gemini Nano was originally limited to the Pixel 8 Pro but arrived on the Pixel 8 and Pixel 8a in June 2024. You can read more about the differences between the three Gemini versions in our explainer at the link.

So what can Pixel phones do with Gemini Nano? Let’s find out!

Gemini Nano feature: Recorder summaries

On the Google Pixel 8 series or newer you can use a Gemini Nano-based feature inside the Recorder app. When you open a transcript of a recording, you will see a new Summarize button. Tapping this will use the power of Gemini to produce a quick transcript summary.

Because this is a Gemini Nano feature, your phone’s processor handles the summary. That means you don’t need to be connected to the internet for the Summarize button to work.

On the Pixel 8 or Pixel 8a, you’ll need the June 2024 Pixel Feature Drop. You’ll also need the newest version of the Recorder app. If you’re using these devices, you must also manually activate Gemini Nano to enable this feature. To do this, head to Settings > About phone and scroll down to the very bottom to find your software build number. Tap this number seven times and then enter your PIN. Then go to Settings > System > Developer options and scroll down until you find AICore Settings. Ensure that all the settings in this menu are enabled.

Gemini Nano feature: Gboard Smart Reply

This feature allows you to receive suggestions for replies within your WhatsApp, Line, and KakaoTalk chats. It works by scanning the most recent chats in a thread and then providing you with a suggested reply that would be appropriate for that moment. Instead of manually typing out this reply, you’d just hit the suggestion, and Gemini will do the work for you.

Obviously, this is not unlike the years-old Smart Reply. But since this uses Gemini Nano, it will work faster, better, and use your phone’s processor rather than the cloud.

To turn this on, you’ll need access to Developer Options. To activate this on your Pixel device, go to Settings > About phone and scroll down to the very bottom to find your software build number. Tap this number seven times and then enter your PIN. Now, go to Settings > System > Developer options and scroll down until you find AICore Settings. Tap this and make sure the Enable AICore Persistent toggle is on. This will give you access to Gboard’s Smart Reply, as powered by Gemini Nano.

Although the Pixel 8 and Pixel 8a also have access to Gemini Nano, Google hasn’t explicitly confirmed that this feature functions on these two devices.

Gemini Nano feature: Magic Compose

Magic Compose essentially suggests message replies for you based on the context of your conversation, which is not that different from Gboard Smart Reply described in the previous section. However, this feature does a lot more and it works exclusively in Google Messages.

In addition to standard message suggestions, Magic Compose can also rewrite messages for you. For example, it could make your message sound more professional, more fun, or even Shakespearean. It can also stack. For example, you could accept a pre-written Magic Compose suggestion and then alter that suggestion to sound Shakespearean. In certain situations, you wouldn’t even need to touch the keyboard letters.

On most Android phones, Magic Compose is already there in Google Messages — but it works in the cloud. The Pixel 8 devices, though, can use Magic Compose features using on-device power, thanks to the power of Gemini Nano. This should make suggestions faster and better.

Notably, with their new Gemini Nano support, the Pixel 8 and Pixel 8a could theoretically support this feature. However, Google hasn’t explicitly stated that the feature is available on either.

Gemini Nano feature: Call Notes

Google increased the amount of RAM on all Pixel 9 series devices to accomodate Gemini Nano and other AI models, allowing them to run in the background. This has opened up new features like Call Assist, which uses Gemini Nano to create an AI-generated call summary for regular phone calls. You can either view the transcription in full or access a summarized version via the Pixel Dialer app.

Call Notes runs entirely on-device, so you don’t have to worry about your phone calls being accessed by anyone else. You will have to enable Call Notes on a per-call basis and it will notify both parties that the conversation is being recorded.

Gemini Nano feature: Pixel Screenshots

The Pixel Screenshots app is a new app exclusively available on the Pixel 9 series. The app takes advantage of Gemini Nano’s multimodal nature, allowing it to scan through the contents of any screenshots you capture. The search function within the app can then be used to narrow down what you’re looking for, including text and images within the screenshot. The app can also create collections, like the boots example pictured above.

Since the Pixel Screenshots app uses an on-device AI model, it works entirely offline and doesn’t leak any of your information to the cloud. The model can also answer questions based on your search term. For example, typing in “t-shirt price” will yield a response along the lines of “The t-shirt costs $39” after analyzing your screenshots.

Imagen 3 feature: Pixel Studio

With the Pixel 9 series, Google introduced a new app exclusively for AI image generation. Dubbed Pixel Studio, it relies on the company’s Imagen 3 model in the cloud but uses the Tensor G4 for on-device edits. This announcement came just weeks after Apple announced a similar Image Playground app for the iPhone. Once Pixel Studio generates its images, you can edit them with a handful of in-app options like the ability to add your own photos and overlay text.

The Pixel Studio app won’t produce photorealistic results like Midjourney or any of the other top AI image generators, but it will get the job done if you just need stickers for your chats. You can also bring up the Gemini chatbot and ask it to create images without the Pixel Studio app, but the downside is that you won’t be able to edit photos after the fact.

Gemini Pro is also working in Gemini and Gemini Advanced

On December 6, 2023, Google formally switched on Gemini Pro inside its chatbot then known as Google Bard. According to Google, this made Bard “far more capable at things like understanding and summarizing, reasoning, coding, and planning.” Later, Google did away with the Bard branding and changed it to…Gemini. That means Gemini (the chatbot) runs on Gemini (the LLM). We know this is confusing, but Google is Google.

If you communicate with the Gemini chatbot or its paid version, known as Gemini Advanced, you will use Gemini Pro, the cloud-based version of the Gemini LLM. While this technically isn’t a Pixel feature, you can access Gemini or Gemini Advanced on a Pixel through your browser. You can try it out for yourself at Gemini’s web portal. You can also replace Google Assistant on your phone with Gemini, which allows you to call the service in the same ways as you currently call on Assistant.

Gemini-based features we hope to see for Pixel phones soon

We’ve only seen a few features integrating Gemini Nano and Gemini Pro, but there’s undoubtedly much more on the way. Below, we have a few ideas as to cool new tricks Pixel phones could do. To be clear, these are not rumored features or things Google has told us might be on the way — these are just ideas for what we hope to see.

Photography and editing features

Google already uses a lot of AI for photography, videography, and media editing on Pixels. These tools help Pixels to be some of the best camera phones you can buy. Gemini could take these features up a notch and produce photography and videography the likes of which we’ve never seen from a phone.

As an example of what this could mean, the Pixel 8 Pro introduced a feature called Video Boost. This processes each frame of a video using the same principles behind the processing a Pixel does to a captured photo. To use Video Boost, though, you need to shoot that video on the Pixel 8 Pro, upload it to Google, wait for Google to process it, and then download it. Maybe, at some point in the future, this process could happen on-device with the power of Gemini Nano. Going further, in the distant future, this could maybe even happen in real-time as you record the clip. Still, it has improved and become faster on the Pixel 9 Pro.

Likewise, we already have Magic Editor for Pixel phones (and the free version on other Android devices). Right now, though, it’s pretty simplistic and also needs a data connection. With Gemini Nano, Magic Editor could become the next Photoshop but be so easy to use that anyone could do it.

Given how popular Pixels are for photos and videos, we are certain Google has a lot of big plans for Gemini in this realm.

Complex mapping

Google Maps has a ton of capabilities already when it comes to mapping and directions. However, even on desktop, you can’t get too complex with it. For example, you can’t incorporate variables in your route like weather reports, concerts or other events that could cause significant spikes in traffic, or your own personal driving habits.

With Gemini, though, these variables could all come into play. You could request directions to a location, and Google could pull in all those different factors to give you the ideal route. It could even give you a hyper-accurate window of time for your arrival.

With Gemini now replacing Google Assistant in many respects, you could even make the creation of complex navigation completely natural. For example, you could say, “Hey Google, I need directions to work that will bypass Route 101 and get me there by my usual time.” That’s a command that is way too complex for Maps/Assistant to understand now, but Gemini could make it possible.

Quick productivity

One thing that we’re starting to see the Gemini chatbot be able to do is create tables in Sheets, papers in Docs, and emails in Gmail using only voice prompts. With more time, though, this could become incredibly powerful.

Imagine being able to record a meeting with Recorder and then ask Gemini to create a table of all the major data points discussed. Or, imagine having Gmail automatically unsubscribe you from any email list that you haven’t opened in the past 90 days. You could even draft a report in Docs and then ask for it to be formatted in a specific style, like Chicago-style or Associated Press Stylebook, which could save you a ton of time.

What Gemini features are you most excited to see on Pixel? What features are you hoping to see in the future? Let us know in the comments!