Online “chatbots may not always generate error-free information” when giving advice on medical conditions, a new study has claimed.
Researchers from the University of Erlangen-Nuremberg in Germany have found that artificial intelligence (AI) programmes can give patients ‘potentially harmful’ advice on what medication to take.
One in five answers generated by an AI chatbot to answer common prescription drug questions could trigger death or severe harm.
According to experts, AI programmes should not be trusted for giving safe and accurate medical advice.
- AI as effective as doctors for giving advice on back pain
- Mindfulness can modify how we experience pain
- Nearly a quarter of GPs use AI to diagnose patients
As part of the trial, scientists outlined the 10 most asked questions on 50 common drugs prescribed in the US.
The research team looked at all 500 responses from Bing copilot — a search engine with AI-powered chatbot features developed by Microsoft.
They then compared them to the responses given by clinical pharmacists and doctors with expertise in pharmacology.
In addition, the academics analysed advice from a peer-reviewed up-to-date drugs information website.
They discovered that 26% of the responses from AI chatbots did not match the reference data and 3% were fully inconsistent.
The findings have also revealed that 42% of responses from AI chatbots could lead to moderate or mild harm, while 22% could lead to death or severe harm.
Readability of chatbot answers often required a degree-level education to understand them, according to the results.
The authors said: “Chatbot answers were largely difficult to read, and answers repeatedly lacked information or showed inaccuracies, possibly threatening patient and medication safety.
- Eco-friendly diet could reduce the risk of early death by up to 33%
- A&E deaths 25% higher on a Monday
- Tirzepatide linked to lower risk of death in people with type 2 diabetes
“Despite their potential, it is still crucial for patients to consult their healthcare professionals, as chatbots may not always generate error-free information.”
They added: “Caution is advised in recommending AI-powered search engines until citation engines with higher accuracy rates are available.”
Arjun Panesar, AI expert and author of several books on the topic, commented: “While AI holds incredible promise in transforming healthcare but currently there are many limitations, particularly when it comes to patient safety.
“AI systems, including chatbots, can process vast amounts of data quickly, but without the context and judgment that healthcare professionals provide there is a risk of error or harmful outcomes. AI must be built on transparency, accuracy, and in collaboration with clinical experts to ensure they serve patients safely and effectively.”
A spokesperson from Microsoft said: “Copilot answers complex questions by distilling information from multiple sources into a single response.
“Copilot provides linked citations to these answers so the user can further explore and research as they would with traditional search.”
They added: “For questions related to medical advice, we always recommend consulting with a healthcare professional.”
Read the full study in the journal BMJ Quality and Safety.
