AI and Mental Health: Red Flag Laws are Not the Answer

I confess -- I was anxious before my first Waymo ride. Getting into a driverless car felt like a high-risk experiment. Ironically, as Louis Anslow noted in a wonderful essay, when we first switched from horse-drawn to horseless carriages, the anxiety ran the other way — the danger wasn’t the machine, it was the human behind the wheel. In 1896 Alfred Sennett warned, “We should not overlook the fact that the driving of a horseless carriage calls for a larger amount of attention for he has not the advantage of the intelligence of the horse in shaping his path.” To manage this risk, the Red Flag Act in England required that a person waving a red flag walk in front of a horseless carriage to reduce the speed and ensure the safety of both driver and pedestrians. The law made early automobiles absurdly impractical — until it was repealed, recognizing that progress requires better safeguards, not paralysis.

Maybe it’s worth recounting this history as we consider regulation of AI in mental healthcare. So far, Illinois and Nevada have banned the use of AI for diagnosis or therapy of mental disorders. Bills introduced in Massachusetts, New York, and Pennsylvania are following suit. Recent hearings in the House and Senate have extolled the use of AI for healthcare and fretted over the use of AI for mental healthcare. As Sen. Josh Hawley (Rep, Missouri) asked his colleagues, “Doesn’t Congress need to do something now in order to protect kids from being killed by AI systems?” And on Thursday of this week, unless the government shutdown intervenes, the FDA’s Digital Health Advisory Committee will discuss and make recommendations on AI-enabled digital mental health medical devices.

Even the industry has been talking about how to manage the risks. On Oct 26th, Open AI published, “Strengthening ChatGPT’s responses in sensitive conversations”, with a subtitle: “We worked with more than 170 mental health experts to help ChatGPT more reliably recognize signs of distress, respond with care, and guide people toward real-world support – reducing responses that fall short of our desired behavior by 65-80%.” This post included some remarkable data about ChatGPT’s users:

· Mental health emergencies related to psychosis or mania: 0.07% users/week

· Explicit indicators of potential suicidal planning or intent: 0.15% users/week

· Potentially heightened levels of emotional attachment to ChatGPT: 0.15% users/week

These apparently tiny percentages look different if you realize that OpenAI estimates 800M users of ChatGPT each week. That is… 560K people with psychosis, 1.2M people with suicidal intent, and 1.2M with over-dependence. Each week. A spate of gut-wrenching stories has documented suicides and psychosis in people who have been using GenAI products like ChatGPT. It’s reassuring to know that OpenAI has responded by improving their products, reducing adverse responses by 65-80%. That leaves 20-35% of 1.2M suicidal people, or 240-420K people, still at risk. Each week. And, of course, don’t forget that ChatGPT estimates that many millions (perhaps 49% of users) use the app each week for psychological support and guidance.

That sounds alarming — and it is — but banning AI in mental health care is the digital equivalent of waving a red flag in front of a horseless carriage. It slows progress without making anyone safer. These bans do nothing to reduce the risk for people using ChatGPT and other foundational AI models. None of these were developed for diagnosis or therapy of mental disorders and none purports to offer diagnosis or therapy. Indeed, while these tools are masterfully engaging and even seductive, for each of these companies, mental illness is a bug, not a feature of their business model. They want this problem to go away.

But there is a cohort of start-ups that are developing AI models to focus on diagnosis and treatment of mental illness, just as other start-ups are using these tools to improve our approach to cancer, heart disease, and diabetes. We need innovation for mental healthcare. AI tools developed specifically for mental healthcare could address the huge supply-demand mismatch and possibly improve the quality of care. The right path isn’t prohibition – it’s guardrails for the safe development of AI.

What are those guardrails? Several models have been suggested (see, for example, Tavory and this open letter). The basics include (at least) the following:

• Clear disclosure that the “helper” is AI, not a licensed clinician

• Built-in redirection to human crisis services

• Design features that limit overuse, especially for youth

• Real-time monitoring for high-risk behaviors and clinical outcomes

• Transparent data-use and privacy commitments

Would these matter for those at-risk 240-420K people using ChatGPT this week? Maybe not. Maybe we need what Tavory calls a “defined duty of care.” That is, a commitment to effective care, a responsibility for outcomes, and an accountability for adverse events. In last month’s Senate hearing, one of the witnesses, John Bailey of the American Enterprise Institute, made an interesting suggestion. Pointing out that AI companies develop benchmarks for accuracy, speed, energy use, he wondered if there could and should be benchmarks for well-being.

I’ve had many Waymo rides since that first anxious moment. Some of them have been imperfect (one Waymo had to circle the block a few times before getting the right route), but nearly all have been smoother and safer than their human-driven equivalents. Mental-health care isn’t transportation, but it’s another system failing to meet demand. Millions already turn to ChatGPT for psychological support because they can’t find or afford a therapist. The solution to that gap isn’t to outlaw innovation; it’s to guide it responsibly.

More than a century ago, we learned that progress behind a red flag goes nowhere. Let’s not make the same mistake with innovation in mental healthcare.