Comparative analysis of token classification and zero-shot LLM approaches for Named Entity Recognition in the business domain

Named Entity Recognition (NER) is a critical task in natural language processing with significant implications for various downstream applications. This paper presents a comprehensive comparative study of NER performance across multiple domains, focusing on the business-oriented BUSTER dataset and the widely used CoNLL 2003 English dataset. We evaluate a diverse set of models, including fine-tuned Question Answering (QA) models such as BERT and RoBERTa, as well as state-of-the-art generative Large Language Models such as GPT series 3 and 4 [1], LLaMA 2-3 [2], [3] and Mistral [4].Our study investigates the adaptation of QA models for NER and examines the zero-shot capabilities of generative models, assessing their intrinsic ability to identify named entities without task-specific fine-tuning. Through extensive experimentation, we analyze precision, recall, and F1 scores across different entity categories, comparing performance across datasets and model families. Additionally, for QA models we explore the robustness of these models under different training setups and evaluation metrics, shedding light on their adaptability to structured and unstructured text data. Our findings provide insights into the effectiveness of both fine-tuned and zero-shot approaches, with models achieving state-of-the-art performance through fine-tuning. This contributes to a broader understanding of the NER task, particularly in domain-specific contexts.