Performance of Large Language Models on Iran’s Medical Informatics Graduate Entrance Exams

Abstract

 Large language models (LLMs) with advanced natural language processing are increasingly used in medical education. This study evaluated and compared the accuracy of four LLMs (GPT-4O, O3-mini, Gemini, and Copilot) in answering questions from Iran’s master’s and doctoral entrance exams in medical informatics. Multiple-choice questions from the 2024 exams, 116 for master’s and 96 for doctoral programs, were submitted to the free versions of the models using uniform prompts. Responses were compared with the official answer key to measure accuracy, uncertainty, and error rates. Statistical analyses included Chi-Square tests and logistic regression. At the master’s level, O3-mini performed best, while Copilot was weakest, though the differences were not significant (p = 0.2088). At the doctoral level, GPT-4O and O3-mini outperformed Gemini, which had 79.44% accuracy and a 21.55% error rate, with significant differences (p < 0.001). Model performance varied across specialized subjects. These results indicate that LLM performance depends on model type, educational level, and the nature of the content, providing a foundation for more accurate assessments and targeted AI applications in specialized education.