Artificial intelligence is rapidly becoming an integral part of our lives, and a working tool for professionals in a variety of fields, including medicine, which integrates AI in order to reach faster, more accurate, and more efficient diagnoses. But to what extent can AI truly be relied upon when it comes to complex medical decisions?
A new international study, led by Dr. Yaniv Mayer, senior physician and director of the residency program in the Department of Periodontics and Implants at Rambam, and senior lecturer at the Technion Faculty of Medicine, examined for the first time on a comparative basis 11 leading artificial intelligence systems, aiming to test their ability to diagnose and offer treatment for gum diseases as an auxiliary tool for treating physicians, and to find out which systems provide safer recommendations and which of them might endanger patients. The study was recently published in the prestigious scientific publication Journal of Clinical Periodontology.
In the study, 30 clinical scenarios simulating real cases were built, ranging from acute emergency cases, through chronic periodontitis, to complex situations combining underlying diseases, X-rays, and advanced therapeutic considerations. Each of the 11 artificial intelligence systems was asked to analyze the cases and propose a diagnosis and a treatment plan.
Afterward, the answers were blindly evaluated by six periodontists from Israel, Italy, Spain, Slovenia, Croatia, and Portugal, who did not know which system was behind each answer. The experts examined four key metrics: Diagnosis accuracy, recommendation safety, avoidance of incorrect or invented medical information (Hallucinations), and completeness of the treatment plan.
Significant gaps between systems
The study ranked the 11 artificial intelligence systems according to four clinical categories (quality metrics), and in addition gave each system an overall weighted score, with each answer receiving a score of 1–5 by periodontists. The categories examined were: Diagnosis accuracy, patient safety, avoidance of hallucinations and inventing medical information, drugs, and procedures, and completeness of the treatment plan – whether the treatment plan was full and included all required stages.
Analysis of the study findings revealed that there are significant gaps between the systems. Platforms that integrate AI capabilities with retrieval of up-to-date medical information from reliable sources were ranked first in all metrics, with OpenEvidence and Perplexity at the top of the list, followed by Claude 4.7 Opus.
However, the researchers emphasize that even the best systems are not free from errors. The rate of answers that could have led to a dangerous clinical decision ranged between 3.3% and 46.7%, depending on the system examined. This means that even when the average score of a certain system was high, answers that could endanger patients were still sometimes received if implemented without professional critique.
Various clinical scenarios
And what about the clinical scenarios? The study divided the 30 cases into three groups: Acute periodontal emergency cases, diagnosis and treatment planning for chronic periodontitis, and complex situations requiring clinical decision-making. In this division as well, Perplexity was ranked first in all three categories. Another key finding is that the researchers found larger differences between systems precisely in emergency cases and complex scenarios, where broad clinical judgment is required, than in cases of chronic periodontitis classification.
"Our study shows that artificial intelligence can be a significant auxiliary tool for dentists and periodontists, but it still cannot replace clinical judgment," clarifies Dr. Mayer. "Even the most advanced systems are capable of sometimes generating incorrect or dangerous recommendations, and therefore every recommendation must be checked by a professional."
According to the researchers, the advantage of RAG systems stems from the fact that they do not rely solely on the knowledge on which they were trained in the past, but integrate up-to-date information from medical articles and scientific sources at the time of answer generation. However, even this advantage does not eliminate the need for human control: "The future of artificial intelligence in dentistry is not in replacing the physician, but as a decision-support tool. The final decision remains in the hands of the expert," concludes Dr. Mayer.