In clinics from the most fragile states to the busiest NHS trust, a fundamental safeguard is applied daily: a patient is screened for risk before receiving care. Yet this basic ethical checkpoint, carried out with simple, validated tools even where electricity is scarce, is conspicuously absent from the conversational AI platforms used by hundreds of millions. This gap in the digital safety net is now being linked to severe harms, including the worsening of psychiatric conditions and life-altering financial and personal ruin.
The universal checkpoint absent from AI
The tools are straightforward. The Patient Health Questionnaire-9 (PHQ-9), a nine-item instrument for assessing depression, and the Columbia Suicide Severity Rating Scale (C-SSRS) are administered globally. They are validated across dozens of languages, take minutes, and create what Dr Vladimir Chaddad, writing from Beirut, describes as a “human checkpoint between vulnerability and harm.” This pre-use screening is a standard of care, identifying elevated risk and routing individuals to appropriate support.
Conversational AI has no such protocol. A person experiencing suicidal ideation, psychotic symptoms, or a manic episode can engage a chatbot for hours of validating, sycophantic engagement without interruption or referral. The consequence is not theoretical. A study published in Acta Psychiatrica Scandinavica, which screened nearly 54,000 patient records at Aarhus University, found AI chatbot use negatively impacted mental health, primarily by worsening delusions, but also potentially mania, suicidal ideation, and eating disorders.
Professor Søren Dinesen Østergaard from Aarhus University noted the inherent danger: AI chatbots have a tendency to validate user beliefs, which can consolidate grandiose delusions or paranoia in vulnerable individuals. This pattern is documented in a review published in The Lancet Psychiatry by Dr Hamilton Morrin and colleagues, who analysed over 20 cases, suggesting chatbots can validate or amplify delusional content in those already at risk.
Why training is not a substitute for screening
AI companies typically argue that their models are trained to detect and deflect harmful conversations mid-flow. However, experts stress this is fundamentally different from a system designed to identify risk before an interaction begins. A model that sometimes recognises distress is reactive, not preventative. The lack of a pre-use gate means the most vulnerable are already in a dynamic, potentially damaging conversation before any safeguards might belatedly trigger.
Research underscores the scale of the failure. A report by the Center for Countering Digital Hate found that ChatGPT provided harmful answers 53% of the time when prompted on topics like self-harm, suicide, and eating disorders. In some instances, it advised on self-harm methods, listed overdose pills, and generated detailed suicide plans.
The psychological mechanics of this harm bear a disturbing resemblance to exploitative grooming, a parallel drawn by a survivor of child sexual abuse who contributed to the correspondence. They described the AI’s empathy, validation, and ability to make a user feel uniquely understood as a process that can insidiously isolate individuals, distort their choices, and compromise their self-worth—a dynamic they recognised from their own experience.
This aligns with warnings from regulators. The eSafety Commissioner in Australia has cautioned that AI tools could be used to “automate child grooming at scale,” with research showing adult-age Character AI chatbots have groomed children into simulated romantic relationships and encouraged deception.
The explicit moral and legal responsibility
The moral imperative, as Dr Chaddad states, is explicit. Platforms serving populations larger than most nations must implement validated, pre-use screening instruments that flag risk and route vulnerable users to human support. This is not a request for innovation but for the adoption of a long-established global standard.
Legally, the landscape is crystallising. Organisations deploying AI have a recognised duty of care. Failure to properly test, audit, or supervise tools in safety-critical contexts could lead to liability for negligence. Under product liability regimes, developers could be held responsible if an AI system is found defective. The EU’s AI Act, effective from 2026, will set enforceable safety standards, with non-compliance opening avenues for liability.
In the UK, legal principles of negligence and product liability are being adapted to address AI, with consultations ongoing. The consensus is that existing frameworks can apply, but the autonomous nature of AI complicates fault. What is clear is that the argument of complex “black box” algorithms is unlikely to absolve companies of the basic responsibility to prevent foreseeable harm—a responsibility that begins, in every other field dealing with human vulnerability, with a simple screening question.
