Google Gemini Goes Evil, Tells Student ‘You’re A Burden, Please Die’

Fears about AI rising up and annihilating humans are a bit premature at this point, but neural networks can still behave in some unpredictable and shocking ways at times. Case in point: Google's Gemini AI chatbot just unsubtly told a human to die—but at least it was polite enough to say "please" first.

Jokes aside, it really happened. The Gemini conversation link is still up, and available to look at on Google's website if you're interested. Here's a screenshot of the full response that the AI gave to Vidhay Reddy, a 29-year-old student who had been asking the chatbot for help with his homework:


The Agent Smith-like response came out of nowhere, in response to a confusing prompt that seems to have been Vidhay putting his homework into the AI and hoping it gave him the answer. Discussions about the ethics of this kind of use of AI aside, there was certainly nothing in Vidhay's prompt that should have inspired Gemini to respond this way, but generative AI can be unpredictable.

Many users will have had the experience of using an AI image generator only for the final output image to have elements in it that were not part of the prompt, or for the image to be wrong altogether. This is essentially the same thing that has happened here, as Google said in its response to CBS News when asked about the incident:

Large language models can sometimes respond with non-sensical responses, and this is an example of that. This response violated our policies and we've taken action to prevent similar outputs from occurring.
— Gemini creator Google, to CBS News

Even still, the very personal and specific nature of Gemini's response is remarkably dark and disturbing. Vidhay and his sister described the message as having potentially fatal consequences for someone who was "alone and in a bad mental place." We're not sure if we would blame that on the language model directly, but in any case it's certainly not a horrific response.