Can AI improve triage quality? A preliminary assessment of ChatGPT performance in evaluating triage decisions

his study explored the potential role of ChatGPT as a retrospective evaluator of triage appropriateness according to the Tuscan Triage System. Fifty real-world Emergency Department clinical scenarios were independently assessed by certified triage experts and compared with evaluations generated by ChatGPT (GPT-4o). Agreement between the model and the expert-defined reference standard was analyzed using Cohen’s kappa, precision, recall, F1-score, sensitivity, and specificity. Exact agreement was observed in 46% of cases, with under-triage discrepancies occurring more frequently than over-triage. Performance was higher in high-complexity scenarios, while lower agreement was observed in moderate- and low-complexity cases. Although the current results do not support autonomous use of large language models for clinical triage, the findings suggest a potential future role in retrospective quality assurance and audit processes. Further studies are required to improve performance and validate the application of artificial intelligence tools in emergency nursing practice.