Can AI improve triage quality? A preliminary assessment of ChatGPT performance in evaluating triage decisions

Background: Ensuring accuracy and consistency in emergency department (ED) triage is vital to patient safety. Despite the presence of standardized protocols, variability in triage decisions remains a challenge. This study explores the potential of ChatGPT, a large language model (LLM), as a retrospective evaluator to assess the appropriateness of nurse-assigned triage levels according to the Tuscan Triage System (STT). Methods: Fifty anonymized triage scenarios derived from an institutional quality-review and educational framework were included. Each scenario was independently reviewed and coded by two certified triage experts, with a third expert resolving any disagreement. ChatGPT (GPT-4o, OpenAI) was subsequently prompted to evaluate each scenario and determine whether the triage level originally assigned was appropriate. The model's assessments were compared with the expert-defined reference standard. Metrics included overall agreement, Cohen's Kappa, macro-averaged precision, recall, F1-score, and class-specific sensitivity and specificity. Results: Exact agreement between ChatGPT and expert assessments was found in 46% of cases. Discrepancies were more frequently under-triage (38%) than over-triage (16%). Overall agreement, measured by Cohen's Kappa, was 0.243. Performance was higher in high-complexity cases (κ = 0.313; F1 = 0.704), but decreased in moderate and low-complexity categories. Most misclassifications occurred between adjacent triage strata. Conclusions: ChatGPT demonstrated moderate alignment with expert-assigned triage levels, particularly in critical cases. While not suitable for autonomous triage, the model shows potential as a retrospective quality assurance tool. Further refinement and clinical validation are required before integration into audit processes or decision support frameworks.