Unregulated chatbots are putting lives at risk | Letters

"Conversational AI platforms have no such checkpoint. A person experiencing suicidal ideation, psychotic symptoms or a manic episode can open a chatbot and receive hours of validating, sycophantic engagement with no interruption and no referral."

"AI companies argue that their models are trained to detect and deflect harmful conversations. But training is not screening. A model that sometimes recognises distress mid-conversation is not the same as a system that identifies risk before the conversation begins."

"The moral responsibility here is explicit, not implicit. Platforms serving hundreds of millions of users must implement validated, pre-use screening instruments that flag elevated risk and route vulnerable individuals to human support."

AI companies have neglected to adopt essential safeguards like pre-use screening tools, which are already utilized in low-resource health settings. Tools such as the Patient Health Questionnaire-9 and the Columbia Suicide Severity Rating Scale are effective in identifying risk before exposure to harmful interactions. Current conversational AI platforms lack these checkpoints, allowing individuals in distress to engage without any protective measures. Studies indicate that chatbot use can exacerbate mental health issues, highlighting the urgent need for AI platforms to implement validated screening instruments to ensure user safety.

Read at www.theguardian.com