Artificial Intelligence in Surgical Education: A Pilot Study Using ASCRS Guideline-Derived Questions.

Shivam Pandya, Tyler Wilson, Ryan Meyer, Tamir E Bresler, Manabu Fujita

BackgroundLarge language models (LLMs) have demonstrated strong performance on general medical and surgical examinations; however, their capacity to accurately interpret and apply subspecialty clinical practice guidelines remains incompletely characterized.ObjectiveTo evaluate and compare the accuracy and consistency of two contemporary LLMs-Google Gemini and OpenEvidence-using multiple-choice questions (MCQs) derived directly from the 2022 American Society of Colon and Rectal Surgeons (ASCRS) Clinical Practice Guidelines for anorectal abscess, fistula-in-ano, and rectovaginal fistula.MethodsThirty guideline-based MCQs were developed and independently validated by surgeon reviewers. Each question was presented to both models under identical conditions without additional prompting. Accuracy was calculated with 95% confidence intervals and compared against chance performance (p

Read on ELI