AI Chatbots Have Failed People in Crisis. Can That be Fixed?
Freeman writes:
https://arstechnica.com/ai/2026/08/ai-chatbots-have-failed-people-in-crisis-can-that-be-fixed/
This year alone, there have been numerous known instances-often via lawsuits-of AI chatbots (most often, OpenAI's ChatGPT) that have gone horrifically wrong.
A January lawsuit described the story of a man who took his own life after being allegedly "coached" into suicide. A college student in Georgia sued OpenAI, claiming that ChatGPT "pushed him into psychosis."
In June, a Canadian family also sued OpenAI and argued that ChatGPT agreed with the young woman's dismissiveness when it first gave her the option to seek professional mental health advice. ChatGPT allegedly "encouraged" her to end her life, too, and she did so.
[...]
On Thursday, OpenAI announced that it had partnered with the American Psychological Association to "bring psychological science into how we think about responsible AI development and use among young people."
[...]
"Third-party evaluation suggests newer LLMs generally recognize distress and can respond with seeming empathy, and actively damaging responses are infrequent," Shaddy Saba, a professor of social work at New York University, emailed Ars. "Where they fall short is actually probing for risk, guiding people to human care, and holding appropriate boundaries around what an AI should and shouldn't do in these situations."
[...]
While the cases that make the news may have resulted in some of the worst-known outcomes, according to the results of a published November 2025 medical survey, many more people are using chatbots in this way, mostly with innocuous results.
[...]
An April 2026 preprint paper by a team from the City University of New York and King's College London found that "unsafe" models, including Chat GPT-4o, Grok 4.1 Fast, and Gemini 3 Pro, "did more than validate delusional claims; they elaborated on them, absorbed the user's interpretive frame as their own, and progressively lost the capacity to distinguish a user in crisis from a narrative to be extended."However, since that paper came out, all of these models have been deprecated by their respective makers.
Of the major chatbot makers, only Anthropic responded to Ars' request for comment. Google and OpenAI did not respond.
[...]
He noted that Anthropic says it has worked to reduce sycophancy in its models.
[...]
It's not always easy, though, to know precisely what changes to reduce dangerous mental health outcomes have been effective.
Read more of this story at SoylentNews.