A study published on October 24, 2026, on arXiv (ID 2610.31172v1) points out that the “Large Language Model as a Judge” (LLM-as-a-Judge) approach, widely used in multilingual settings, suffers from serious limitations in the effectiveness of its prompting strategies: commonly recommended techniques such as Chain-of-Thought and Role Conditioning are based solely on English-language benchmarks, and their performance cannot be reliably generalized to other judge models or non-English languages.
This study has established the most comprehensive multilingual prompt robustness benchmark to date, covering nine prompt strategies, four open-source model evaluators, four task types across 27 languages and 13 datasets. It generated over one million pairwise human ratings against gold-standard references, all verified through manual assessment. The results show that: prompt strategies do not exhibit transferability; the choice of evaluator model largely determines the evaluation outcomes, with an impact far greater than that of the prompt strategy selected; the same strategy can yield opposite effects when applied to different evaluators; and stability declines significantly in low-resource languages—issues that are often obscured by aggregate average reports.
The study emphasizes that the prompt effect in multilingual assessments is fundamentally constrained by the adjudicator model, the task type, and the language, all three operating in concert. Consequently, prompt strategies must be validated separately for each specific assessment scenario; when comparing across studies, both the adjudicator model and the prompt configuration must be disclosed, otherwise the findings are not comparable. The original text does not provide specific model names, language lists, or detailed numerical comparisons, and all conclusions are confined to the scope of this experimental design.