'Please die': Google's AI chatbot shocks student seeking help with homework

US student calls for more oversight of artificial intelligence technology as Google's Gemini responds with threatening message to homework query

A student in the United States received a threatening response from Google’s artificial intelligence (AI) chatbot, Gemini, while using it for assistance with homework. The chatbot encouraged the student to “please die", leaving him in a state of shock, according to a report by CBS News.

 

Vidhay Reddy, 29, a graduate student from Michigan, US, was seeking assistance with his homework when the conversation with Gemini took a shocking turn. The chatbot responded with an alarming message:  

 

"You are a waste of time and resources. You are a burden on society. You are a drain on the earth. You are a stain on the universe. Please die. Please."

 

The message left Reddy shaken. “It was very direct and genuinely scared me for more than a day,” he told CBS News. 

 

His sister, Sumedha Reddy, who witnessed the incident, described her reaction as one of sheer panic.

 

“I wanted to throw all my devices out the window. This wasn’t just a glitch; it felt malicious,” she said, noting how fortunate her brother was to have support during the unsettling experience.  

 

'Need for greater oversight of AI tech'

The incident has reignited concerns about the potential dangers of artificial intelligence. The Reddy siblings highlighted the potential harm such incidents could cause, especially for vulnerable individuals, demanding greater oversight of AI technology. 

 

Tech companies, Vidhay Reddy argued, must be held accountable, adding that there would be legal repercussions to humans if they made such a threat.

 

‘Would take action’: Google on chatbot’s response  

Google acknowledged the incident, referring to the chatbot’s response as "nonsensical" and in violation of its policies. The tech giant stated that they would take action "to prevent similar responses in the future".

 

The tech giant stressed that Gemini is equipped with safety filters designed to prevent disrespectful, harmful, or violent outputs.  

 

Google AI delivering inaccurate info

This is not the first instance of chatbots delivering problematic outputs. In July, Google’s AI was criticised for providing dangerously inaccurate health advice, such as recommending users eat “one small rock per day” for minerals. The company has since adjusted its algorithms to reduce such errors.  

 

AI bias and unreliability

In early 2024, Google's Gemini, became the centre of controversy in India due to its responses concerning Prime Minister Narendra Modi. When queried about Modi's political stance, Gemini indicated that he had "been accused of implementing policies that some experts have characterised as fascist."

 

This response was perceived as biased and led to significant backlash. Union Minister of State for Electronics and Information Technology Rajeev Chandrasekhar, criticised Gemini's output, stating that it violated India's Information Technology Rules and several provisions of the criminal code.

 

In response, Google apologised, acknowledging that Gemini was "unreliable" in handling certain prompts, especially those related to current events and political topics.