Gemini prepares to let parents know when kids discuss self-harm

Instead of just worrying about predators online, these days parents need to be concerned about the impact AI might be having on their kids — and now Gemini looks like it’s working on a new tool to keep them informed about some serious red flags.

For as human as they can sometimes feel, AI chatbots are decidedly not, and it’s incredibly important that we temper our interactions with them appropriately. That can be difficult enough for adults, but teens appear especially vulnerable, and all too often we’ve seen that lead to devastating consequences. Earlier this year we looked at the lawsuit OpenAI was facing in regards to a teen’s fatal overdose, and this week Bloomberg reports that a new lawsuit accuses Google Search’s AI Mode of talking to teens about suicide without warning the parents.

It looks like Google has already been thinking about a better way for Gemini to handle such situations, and we’ve spotted some evidence of work towards that effect in version 17.73.15.sa.arm64 of the Google app for Android.

Looking through the update, we notice some new strings:

Google already lets parents manage access to Gemini through Family Link, and puts further filters on Gemini’s output when interacting with minors. But by the looks of these text strings, Google seems to be planning to augment that with explicit warning messages for parents if kids ever start talking to Gemini about self-harm.

Right now, we’re only seeing Google’s work on the child’s side of this — we haven’t yet seen what the messages could look like on the parents’ end. It’s interesting that Gemini appears to be preparing to give kids a heads-up about the notices — we wonder if that might have an unintended chilling effect, stopping teens from sharing mental health struggles in the first place, and further keeping parents out of the loop.

In addition to this in-development parental safeguard, we’ve also found evidence of Gemini working on new time limits for kids.