A Performance-Based Rubric for Generative AI use in Medical Students' Research Tasks: Development and Initial Psychometric Evaluation.
Nino Shiukashvili, Mariam Rochikashvili, Vasil Kupradze, Nana Gonjilashvili, Nino Gvajaia, Luka Kutchava, Nona Janikashvili, Nino Tevzadze, Archil Undilashvili, Eka Ekaladze
BACKGROUND: As generative AI becomes embedded in medical training, patient safety depends on graduates' ability to recognize AI limitations and bias, document AI involvement transparently, and verify AI-generated information rather than accept it uncritically. We developed a performance-based rubric to assess observable generative AI (LLM) literacy behaviors within authentic coursework. METHODS: In a single-institution evaluation (Spring 2025), third-year medical students ( RESULTS: Mean (SD) domain scores were: AI Use Documentation 0.67 (1.08), Prompt Generation 1.33 (0.69), Verification 0.41 (0.71), and Integration 1.64 (0.67); total score 4.06 (1.80). Floor effects were substantial for AI Use Documentation (64% scored 0) and Verification (60% scored 0). Inter-rater reliability was high (ICC: Documentation 0.99, Prompt Generation 0.84, Verification 0.93, Integration 0.83). Verification was significantly lower than Prompt Generation and Integration (Bonferroni-adjusted CONCLUSIONS: This rubric demonstrated strong scoring reliability and produced initial psychometric evidence consistent with measuring distinct, observable LLM-use competencies. Findings highlight prominent gaps in verification and transparent documentation, reinforcing competency guidance that emphasizes recognizing AI limitations and verifying AI output to protect patient safety. Further multi-site validation and implementation work is warranted.
Read on ELI