Evaluation of LLM Models and AI Engines: Case of Arabic NLP Task – I’rab (Arabic Syntax Analysis)

Abstract

Arabic syntactic analysis (I‘rab) is a challenging task in Arabic Natural Language Processing due to the language’s rich morphology and complex grammatical structures. Although Large Language Models (LLMs) have achieved remarkable success in many NLP tasks, their ability to perform Arabic I‘rab remains insufficiently explored. This work presents a comparative evaluation of several modern LLMs on a benchmark dataset of 450 Arabic sentences covering different sentence types, text sources, and levels of syntactic complexity. An automatic evaluation framework was developed using text normalization, token overlap, TF-IDF similarity, grammatical term coverage, and grammatical conflict detection. The results show significant differences among the evaluated models in terms of accuracy and consistency. Model performance was affected by sentence complexity and source, with generally better results on simple and standard Arabic texts than on complex, poetic, or Qur’anic sentences. Among the evaluated models, Qwen achieved the best overall performance. The study also highlights limitations of strict automatic evaluation, as some correct answers may be penalized because of differences in wording or detail despite conveying the same grammatical meaning. These findings contribute to a better understanding of LLM capabilities in Arabic I‘rab and provide a foundation for future research on automatic evaluation of Arabic linguistic tasks.