Much hyped AI products like ChatGPT can provide medics with ‘harmful’ advice, study says

The much-hyped AI products like ChatGPT may provide medical doctors and health care professionals with information that can aggravate patients’ conditions and lead to serious health consequences, a study suggests.

Researchers considered three nutrition-related diseases in their study, with two experienced dietitians examining a total of 63 ChatGPT outputs of disease management against medical guidelines.

Their findings sound a warning to medics and health personnel who only lean on AI and related products in passing health and medical advice to their patients.

“The research warned against relying solely on AI interfaces for the management of diabetes and metabolic abnormalities and highlighted the need to consult with the dieticians and other members of the health care team,” said Farah Naja, University of Sharjah’s Professor of nutritional epidemiology. “ChatGPT could provide incorrect, incomplete or harmful advice, jeopardizing the quality of medical care and consequently patients’ health and safety.”

Prof. Naja is the lead author of the study published in the European Journal of Clinical Nutrition in which scientists from the United Arab Emirates, Lebanon, Bahrain and Turkey investigate the correctness and accuracy of ChatGPT answers to various prompts for how to manage and prevent diabetes and metabolic abnormalities.

The study covers three major domains: 1) dietary management, 2) nutrition care process and 3) menu planning for a 1,500-calorie diet. For each domain, three prompts were developed, meaning nine prompts per condition.

“A total of 63 prompts were fed into the GPT3.5-turbo0301 model through the ChatGPT interface provided by OpenAI, during October 2023. Two experienced dietitians evaluated the chatbot output’s concordance with the Academy of Nutrition and Dietetics’ guidelines,” the scientists write.

The scientists, according to Prof. Naja, wanted “to raise awareness regarding the role of artificial intelligence in guiding the dietetic care for diseases such as diabetes and metabolic abnormalities.” Their target, she added, was to assess and measure ChatGPT’s accuracy in providing nutritional management for type 2 diabetes, metabolic syndrome, and its components.

“Diabetes and metabolic abnormalities such as high blood glucose and high blood lipids are widespread and their prevalence keeps increasing in most parts of the world, posing significant challenges for individuals, societies, and governments.”

“The management of these diseases requires, in addition to medications, subtle and often complex changes in lifestyle behaviors including dietary intake.”

To evaluate ChatGPT limitations, the researchers selected seven diet-related metabolic diseases due to their public health impact and risk factors. They include type 2 diabetes, metabolic syndrome and its components, namely central obesity, hyperglycemia, hypertension, low levels of high-density lipoprotein, and hypertriglyceridemia.

Prof. Naja pointed to weight loss, which she said was “critical in the management of diabetes and metabolic abnormalities … Yet the outputs of the ChatGPT missed the weight loss recommendations along with guidance on achieving an energy deficit.”

When asked to provide sample menus for the health conditions considered in the study, ChatGPT outputs did not meet the requirements in terms of energy, carbohydrates, and fat, in addition to calcium and vitamin D, Prof. Naja maintained.

Statistics show that diabetes accounts for 2 million out of 17 million people who die annually from a non-communicable disease before age 71 , according to the World Health Organization (WHO).

WHO attributes 19% of global deaths to metabolic risk factors, like overweight and high blood pressure, which are part of the study’s three main nutrition management-related domains.

In their study, the scientists highlight that “the American Heart Association and the American Diabetes Association consider lifestyle modifications, including nutritional management, as the first-line therapy, being 1.59 times more effective in reversing Metabolic Syndrome compared to pharmaceutical treatments.”

The authors acknowledge ChatGPT’s role in nutrition-related health issues, underscoring its “dynamic conversational capabilities” and its potential in offering “personalized and engaging education.”

They also note that the prevalence of an “increasing number of real-world applications of ChatGPT have been launched in the field of nutrition and health, including nutritional counseling platforms, health and fitness apps, public health campaigns and chatbots’ incorporation within school health and nutrition curricula.”

However, they stress that research is scant on whether “the transformative potential of using ChatGPT in nutrition education” and its huge potential powers can be of any “limitations in providing clinical nutritional advice.”

Prof. Naja said the study found that ChatGPT missed “appropriate physical activity and weight loss recommendations along with guidance on achieving an energy deficit” despite their being “critical in the management of diabetes and metabolic abnormalities.

“ChatGPT outputs were incomplete in terms of guidance on specific nutrients … and did not address the need to increase fiber intake or to consume whole grain products for all the considered conditions.”

The study concludes, “ChatGPT, and potentially other future AI chatbots, react to the user’s prompts in ‘a human-like’ way, but cannot replace the dietitians’ expertise and critical judgment.

“While health care practitioners may consult this increasingly available technology for various purposes, they must also be cautious about relying solely on AI chatbots in clinical practice and should collectively raise awareness about associated risks.”