Healthcare Corner: The Dangers of A.I. Health Advice
By Jacob Rueda
DISCLAIMER: The information in this article is not intended to serve as medical advice. Always consult a qualified, licensed clinician or caregiver for guidance on any health concerns or decisions.
Language learning models (LLMs) are becoming a familiar part of everyday health decision making. People will type symptoms into a chatbot prompt and ask things like whether a medication is safe, or if pain might signal a specific condition or disease, among other things.
Joseph Arrington, who holds a Ph.D. in population health sciences from the University of Utah, says the convenience of using LLMs like ChatGPT, Gemini and Copilot puts people at risk of being misinformed about their health.
“Models like ChatGPT currently do not work on deduction and reasoning because you have to be sentient,” he said. LLMs work by predicting text, not by reasoning or operating from a place of conscience.
Arrington said this is where people run into trouble.
“The reason physicians ask us to walk, breathe, speak, or look around is not to simply check off a box on their checklist,” he said, “It is to look for potential comorbidities, or the actual source of the issue that may have been hidden up to that point.” An LLM cannot check vital signs or test reflexes the way a physician can, which makes relying on one for health concerns even riskier.
Once someone enters a prompt into an LLM, it will put out an answer that Arrington said sounds “polished and authoritative” while simultaneously creating “a false sense of safety or even a false sense of alarm.”
According to findings published in March by KFF, formerly known as the Kaiser Family Foundation, 32% of adults are turning to artificial intelligence for health information and advice. The cost of seeing a doctor, along with the convenience of consulting an LLM instead of waiting for an appointment, are among the reasons people use LLMs. Other reasons include embarrassment over discussing sensitive topics, and speed.
One individual who consulted with AI was Paul Winters, a California pastor who later sued OpenAI and CEO Sam Altman after saying ChatGPT downplayed the severity of his medical symptoms and gave him misleading guidance.
Privacy is another concern Arrington raised about LLMs, since they are not HIPAA compliant. HIPAA governs how medical information is stored and shared, but most consumer AI tools are not covered by those rules. Even so, people share sensitive medical information about themselves or loved ones with chatbots every day.
Despite these and other concerns, Arrington said artificial intelligence can play a helpful role in health education, but only when it supports, rather than replaces, the judgment of trained medical professionals.
See the full video below:
Comments