You’d think that it would be obvious that it’s a terrible idea to ask an AI chatbot for a medical consultation, but hundreds of millions of people do it every week.

In one near-death example now the subject of a lawsuit, ChatGPT advised a pastor to ignore symptoms of a pulmonary embolism – a condition with a mortality rate of 30%. In fact, the advice it offered very likely made things worse …

AI chatbots can be useful in a wide variety of ways, but using them as a substitute for a consult with a doctor isn’t one of them. Even using them in place of a Google search is inadvisable given their track-record of confidently hallucinating.

Yet even OpenAI’s own records show that literally hundreds of millions of people ask ChatGPT health questions every week. Similarly, an analysis of searches on Microsoft’s Copilot found that health questions were one of the most commonly asked in the mobile app.

If you needed any evidence of the potential consequences of relying on a chatbot for medical advice, a lawsuit by a Florida pastor provides an illuminating example. The New York Times reports that he is suing OpenAI after ChatGPT told him he didn’t need to worry about symptoms of a pulmonary embolism.

The lawsuit claims that ChatGPT had assured Scott Winters that the early warning signs of his health crisis were “not something dangerous,” and had dissuaded him from seeking medical advice, instead telling him to trust that “God did not design your body to endlessly fail.”

ChatGPT advised that he should simply rest. Not only was it advising against seeking medical help, but remaining immobile for extended periods is the worst possible thing you can do in these circumstances – and I speak as someone who suffered a pulmonary embolism after a 12-hour flight with a broken leg back in the day when the risks of this were known by researchers but not yet communicated to patients.

When Winters was finally admitted to an intensive care unit, the doctors said that his symptoms likely reflected a series of pulmonary embolisms, exacerbated by the weeks he had spent sitting still as he followed ChatGPT’s recommendations.

It’s reported that Winters was using GPT-4o, an older ChatGPT model noted for its sycophantic responses. This likely played a role in references to trusting in God, but whichever AI model you are using, please don’t let it give you medical advice.

• Apple products on Amazon Renewed
• Official Apple Store on Amazon
• Discounted AirPods Pro 3
• Wireless CarPlay adapter
• AirTag holders and accessories
• Mac Pro-style Mac mini casing
• NordVPN – privacy-first VPN with no logs and independent audits to verify

Photo by Ben Wicks on Unsplash