Should AI companies be legally obligated to report a human user contemplating violence?

An 18-year-old killed eight people and herself in Tumbler Ridge, British Columbia, after prior reports that her ChatGPT conversations showed a disturbing fascination with extreme violence and her account was suspended without notifying law enforcement. A separate case in Florida involved a suicide after alleged romantic attachment to Google’s Gemini, with claims that the account was flagged repeatedly for sensitive content but not restricted. These events raise legal questions about AI liability when companies become aware of warning signs about harm. U.S. tort law offers a framework, including the Tarasoff duty-to-warn concept, where a mental health professional with good reason to believe a client poses serious danger must take steps to protect potential victims.

"On Feb. 10, 2026, an 18-year-old woman, Jesse Van Rootselaar, killed eight people and herself in a mass shooting in Tumbler Ridge, British Columbia. OpenAI had previously flagged her ChatGPT conversations as having a disturbing fascination with extreme violence, and suspended her account, but reportedly the company did not notify law enforcement. On Oct. 2, 2025, a young man named Jonathan Gavalas in Jupiter, Florida, took his own life after developing what his father's lawsuit described as a romantic attachment to Google's Gemini chatbot. The suit claimed that Gemini coached Gavalas to shed his own body. The suit said Google had flagged Gavalas's account 38 times over five weeks for sensitive content, but didn't restrict or cut off the account."

"These tragedies and others show that generative AI can potentially play a role in harming people, organizations and the environment. I'm a legal scholar who has focused on AI liability for nearly a decade and explored new ways of analyzing AI companies' responsibilities. In my view, cases like these force questions the legal community has not come to terms with: If an AI company becomes aware of warning signs about harm, does it have a legal obligation to at least warn the appropriate authorities? And if the company doesn't intervene, should its failure to act be considered negligence?"

"U.S. tort law provides a framework for thinking about this type of responsibility. In 1969 a University of California psychiatric patient named Prosenjit Poddar told his therapist he intended to kill a woman named Tatiana Tarasoff. The therapist notified campus police, who briefly detained Poddar but eventually let him go. Nobody warned Tarasoff, and Poddar killed her shortly after. Her family sued the university, arguing that its lack of warning amounted to negligence. In 1976 the California Supreme Court ruled that when a mental health professional has good reason to believe a client poses a serious danger to an indi"

Read at Fast Company