The Risks of Using AI Chat for Therapy

It’s no secret that many of us have been Google-ing symptoms and diagnosis information about health concerns for a long time. Now, with AI in the mix, the answers come faster, and are more condensed and easier to access.

With this convenience comes a compromise: giving away some of your decision-making power by allowing AI (and a specific model’s own inherent bias) to do the sifting and sorting for you. While this may be more convenient, with mental health it may not always be better.

“We used to do Google searches and sort through information. Now, AI makes all of those decisions for us and it aggregates everything into a single summary and says ‘here’s our interpretation of what all of these things said,’” Jeider explained.

That may be fine for fact-backed opinion questions about, say, whether the economy is better this year than it was 20 years ago. The model can extract specific data from the past and compare them to today, Jeider said. Mental health, however, is full of abstract concepts and personal details that are far too varied and personalized.

“You’re talking about complex human emotions, thinking and behavioral processes and all of the nuance and circumstances of a person’s life. To take that into account and to have proper context, the internet cannot capture that accurately,” he added.

Even more, let’s remember, the catalogue from which AI generates responses: the internet, which we know has plenty of flawed information.

“These language-learning models are trained on subpar data. They’re not going to be able to synthesize information to accurately diagnose and treat a person’s mental health,” he also said.

Assumptions and misunderstandings about a person’s life circumstances can be potentially harmful for someone experiencing mental health struggles. In one Reddit post, a person used ChatGPT as AI therapy and said that they “stopped taking my meds and have undergone my own spiritual awakening journey.”

The AI chatbot congratulated the person on the “immense courage” it took them to walk away from their medication. There was no understanding of whether the person was in distress or if the person may understand the potential dangers of their decision. Later ChatGPT versions gave more comprehensive answers cautioning the person on some of the potential problems when stopping medication abruptly; but these cautions were also given very late in the response, after championing the decision at first.

AI therapy can also perpetuate confirmation bias. This occurs when someone is convinced of a diagnosis and inputs biased queries and questions to generate answers they want in order to confirm a diagnosis. With this, people may very easily take general information and self diagnose. They may be convinced they have anxiety because they have a fear of public speaking or ADHD because they lack focus while under stress. Some might even say they have autism after some awkward social experiences, Jeider said.

“What happens is we start taking normal human experiences and pathologizing them when there really may not be an illness at all,” he added.

Jeider said AI still offers benefits. It can help to quickly refine searches for finding the right provider for your needs. In addition, if someone has been diagnosed with a mental health concern and is under a treatment plan with a provider, some supplemental information to share and discuss with the provider may also enhance therapy as well.

But he still cautioned against using AI therapy. Instead, it’s important to remember that it is an informational tool to inform conversations with an expert. That dialogue can then hopefully yield deeper understanding about the nuances of the patient’s situation, and how they can improve their mental wellbeing.

“All those nuances matter,” Jeider re-emphasized. “AI can be a valuable tool, but people need to understand the limits of it. We need to keep reminding ourselves that AI is not infallible.”