Google expands Gemini AI's onscreen awareness capability to more phones

Reportedly, the "Ask about this screen" and "Ask about this video" features of Gemini AI assistant have started to roll out to other eligible Android smartphones

Don't want to miss the best from Business Standard?

Google is reportedly rolling out new Gemini AI assistant features on Android smartphones that allow users to ask Gemini questions related to what’s on their display. Additionally, Gemini is gaining the ability to offer YouTube video summaries and respond to queries related to specific YouTube videos. These features have been available on the latest Pixel 9 series smartphones; however, it seems they are now rolling out to other Android smartphones as well.

Gemini AI’s “Ask about this screen”: How it works

Once the feature is available on your smartphone, open the floating window for the Gemini AI assistant by long pressing the power button. You will see a new “Ask about this screen” floating button on top of the Gemini window. Tapping on the button takes a screenshot of what is on the screen and shares it with Gemini AI.

Once the screenshot is taken, you can ask Gemini questions about objects on your screen. However, it should be noted that since Gemini uses the screenshot to gather display information, it cannot answer questions about the entire page. For example, if you open a news article in a browser and ask Gemini about what is on the display, it will only have access to the information captured in the screenshot, not the entire web page. The feature also does not allow you to capture an extended screenshot.

Gemini AI’s “Ask about this video”: How it works

Similar to the “Ask about this screen” option, Gemini offers an “Ask about this video” option when playing select YouTube videos. The option appears on YouTube videos that have captions, although it works even if captions are not enabled for the video.

Once you select the “Ask about this video” option, Gemini AI can summarise the video for you or answer specific questions about its content. Since the feature works with captioned videos, the visuals in the video are not analysed.