Evaluating the Accuracy of Machine Translation Using BLEU Score: ChatGPT-4 vs Matecat

Abstract

This study aims to qualitatively compare the translation quality of ChatGPT and Matecat from Arabic into English using Bilingual Evaluation Understudy. It specifically seeks to investigate the extent to which ChatGPT-4 and Matecat translations of metaphors align with human translation and whether they can work as a reliable tool for translating metaphors. Matecat scored a BLEU score of 0.36 while ChatGPT-4 scored 0.38. By employing a t-test, the study has revealed that the difference between ChatGPT-4 translation and Matecat translation is marginal and that they both perform in the same way in terms of metaphor translation. Respectively, the data does not support the argument that ChatGPT-4 outperforms Matecat. The two systems’ translations partially align with the human translation, failing to produce a translation that is similar in terms of quality to the human translator as the mean scores of the two systems are below the average.

Highlights:

• ChatGPT-4 and Matecat were evaluated for Arabic–English metaphor translation using BLEU.
• ChatGPT-4 achieved a BLEU score of 0.38, compared with 0.36 for Matecat.
• The difference between ChatGPT-4 and Matecat was not statistically significant.
• Both systems showed only partial alignment with the human reference translation.
• Human translation remained superior, particularly in accuracy, source-text understanding, and sensitivity to target-language norms.