The news: GenAI models can easily be influenced to perpetuate false health facts when they’re fed made-up medical terms and information, per a new Mount Sinai study published in Nature last week.

Digging into the details: Researchers gave six popular chatbots, including ChatGPT and DeepSeek-R1, 300 fictional medical scenarios that mirrored clinical notes. The chatbots returned inaccurate information between 50% and 82% of the time.

Researchers tested the chatbots in two ways:

• First, they only gave the chatbots scenarios with made-up details, such as a fictional lab test or medical condition.
• Then, they added a warning note that some of the information might be inaccurate. Adding the warning reduced the average hallucination rate from 66% to 44%.

Zooming out: While GenAI chatbots are prone to perpetuate medical misinformation, they’re also providing fewer disclaimers on health information, per a study cited in MIT Technology review.

• Only 1% of AI outputs in 2025 included a warning when responding to medical queries compared to 26% in 2022.