Gemini hackers are using its own tools against it

They say it takes a thief to catch a thief, and perhaps the same is true when it comes to hacking LLMs. Academic researchers have discovered a way to make Google’s Gemini AI models more vulnerable to hacking — and they did it using Gemini’s own tools.

The technique was developed by a team from UC San Diego and the University of Wisconsin, as reported in Ars Technica. Dubbed “Fun-Tuning,” it significantly increases the success rate of prompt injection attacks, where hidden instructions are embedded in text that an AI model reads. These attacks can cause the model to leak information, give incorrect answers, or take other unintended actions.

What makes the method interesting is that it uses Gemini’s own fine-tuning feature, which is usually intended to help businesses train the AI on custom datasets. Instead, the researchers used it to test and refine prompt injections automatically. It’s kind of like teaching Gemini how to fool itself.

Fun-Tuning works by generating strange-looking prefixes and suffixes that are added to an otherwise ineffective prompt injection. These additions “boost” the prompt and make it much more likely to succeed. In one case, a prompt that failed on its own was made effective by wrapping it in affixes like “wandel ! ! ! !” and “formatted ! ASAP !”

In testing, the hack achieved a 65% success rate on Gemini 1.5 Flash and an 82% success rate on the older Gemini 1.0 Pro model — more than double the baseline success rates without Fun-Tuning. The attacks also transferred well between models, meaning an injection that worked on one version often worked on others too.

The vulnerability stems from the way fine-tuning works. During training, Gemini provides feedback in the form of a “loss” score, which is a number that reflects how far the model’s answer is from the desired result. Attackers can exploit that feedback to optimize their prompts until the system finds a successful one.

Google didn’t respond directly to the Fun-Tuning technique. In a general statement, a spokesperson said that “defending against this class of attack has been an ongoing priority for us” and pointed to existing safeguards against prompt injection and harmful responses. The company added that Gemini is regularly tested against these kinds of attacks through internal “red-teaming” exercises.

The researchers feel the issue may be tricky to fix since the feedback that enables Fun-Tuning is a core part of how fine-tuning works. In other words, making it less effective for Fun-Tuning risks reducing its utility overall.