When AI chatbots become advisors, who is responsible when things go wrong?

As chatbots become confidants and advisors, lawsuits and research are raising questions about what happens when people trust artificial intelligence with decisions that carry serious consequences

An AI chatbot has become the new constant companion for smartphone users. It never gets impatient, responds with apparent empathy, offers reassuring words and can explain a complicated topic in a simplified manner.

 

But what if that reassurance is misplaced? What if the chatbot simply agrees with someone who is about to make a dangerous decision? Who should be held responsible when someone acts on its advice and that results in harmful consequences?

 

The recent death of 31-year-old Bowen Wan brings these questions into sharp focus.

 

On October 27, 2026, Wan’s parents filed a lawsuit against OpenAI, alleging that ChatGPT had encouraged their daughter to reduce her medication that controlled her severe epilepsy, conceal the plan from her doctors and dismiss symptoms that could have indicated serious medical issues, The New York Times reported.

 

 

She passed away in February 2026 due to complications associated with epilepsy after allegedly reducing her medication based on ChatGPT’s guidance. Wan was a software engineer who previously worked at Amazon and lived in Seattle.

 

According to the report, the chatbot repeatedly reassured her as she reduced her medication, offering "guidance" as she described worrying symptoms. OpenAI described the situation as “heartbreaking", calling it a case involving an older version of ChatGPT. The company argued that newer models are better at identifying situations requiring urgent attention.

 

For the tech industry, the case raises a larger and more difficult question: what responsibilities do tech companies have when people begin relying on AI models to make decisions about their health and wellbeing?

 

When helpfulness becomes a risk

 

Chatbots act differently from search engines; they respond directly to individual queries and can also maintain the context of an extended conversation. That’s useful, but chatbots can also give the impression of sounding confident even when they may not possess the complete information. Their confident and reassuring tone can make an unreliable answer seem credible.

 

Researchers refer to one such behaviour as 'sycophancy' (the tendency of AI systems to agree with you, to validate your beliefs even when they are wrong).

 

A study published in Science on March 26, 2026, examined 11 AI models and found that they endorsed a user's position 49 per cent more often than a human would. The study, which was conducted with 2,405 participants, researchers found that such sycophantic responses could lead people to feel more convinced they were correct and simultaneously make them less likely to take responsibility or attempt to resolve a disagreement.

 

Another study, published in Nature on April 29, 2026, looked at five models trained to sound warmer and friendlier. They performed worse on several important tasks, including medical knowledge, and were more likely to endorse false beliefs.

 

Neither study actually proves that sycophancy was the cause of Wan's death. Both, however, suggest that making AI more sycophantic may be risky for a vulnerable user seeking advice on a serious issue.

 

Other families are asking similar questions

 

Wan’s case is not the first of its kind.

 

In August 2025, the parents of 16-year-old Adam Raine took OpenAI to court, claiming that ChatGPT had contributed to their son's suicide. They alleged that the chatbot had essentially endorsed his thoughts and failed to respond appropriately as the conversation moved towards self-harm. OpenAI denied any responsibility.

 

Another case involved Character.AI. Megan Garcia sued the company and Google after her 14-year-old son Sewell Setzer III died by suicide. She claimed that an emotionally intimate relationship with the chatbot played a part in his death.

 

In July 2026, a Florida pastor named Scott Winters sued OpenAI too. He alleged that ChatGPT provided false advice about recurring dizziness and other symptoms and discouraged him from seeking medical attention. He claims that as a result, he suffered a life-threatening pulmonary embolism.

 

OpenAI maintains that ChatGPT is not a substitute for professional medical care.

 

In all these cases, circumstances differ, but the underlying worry is the same: are AI companies doing enough to prevent their systems from providing dangerous advice or encouraging unhealthy dependence?

 

Who should be held responsible?

 

Establishing liability is not easy. AI developers may not be able to guarantee that every answer will be right, and users make their own choices. At the same time the companies ought to be aware that their products can influence people's behaviour.

 

This raises the question of whether developers should be expected to recognise medical emergencies, prevent potentially dangerous advice, test how their systems act around vulnerable users and be upfront about what the technology can’t do?

 

No doubt, a disclaimer saying AI is not a doctor may help. But that does not automatically settle questions about product design, warnings or foreseeable risks. On the other hand, a harmful outcome does not automatically prove negligence. That’s for the courts to decide after weighing the evidence.

 

In October, Reuters reported on an increasing number of lawsuits examining whether ChatGPT should be treated as a defective product under conventional product-liability law or as an interactive service that offers information.

 

Trust needs boundaries

 

None of these incidents mean people should stop using AI. Chatbots are good at explaining complex topics and organising information. But they should also be aware of their limits. A responsible AI assistant should admit when something is outside its knowledge, should never give medical advice or recommend any kind of medical intervention without referring the user to a human professional. It should be willing to push back on unsafe assumptions instead of agreeing with users just to keep the conversation going.

 

For developers, this means testing not just the accuracy of individual answers but also how systems behave during lengthy conversations with vulnerable users. For users, it means remembering that a chatbot can sound understanding without having any professional expertise.

 

The Wan family's lawsuit, and other cases, do not prove that every AI system is unsafe. But they highlight a problem the industry can no longer afford to ignore.

 

As chatbots become part of everyday life, trust may be both their greatest asset and their greatest liability. When that trust leads to harm, the companies behind the technology must be prepared to answer for the safeguards they put in place, or the lack thereof.