On October 24, 2026, a newly published academic study officially introduced JSFactCheckBench—a fact-checking evaluation benchmark for large language models (LLMs) in Japanese—on the arXiv platform (arXiv:2610.30504v1). This benchmark aims to address a critical shortcoming of existing evaluation frameworks: prior benchmarks have consistently failed to explicitly define three core design elements simultaneously—namely, “the questions that the statements seek to answer,” “the criteria for judging each statement,” and “the information sources recognized as evidence.”
JSFactCheckBench contains 1,913 statements, all derived from LLM responses to misleading information that had previously circulated widely on Japanese social media platforms. Each statement was subjected to dual human annotation—based on pre-selected online sources—at a specific reference time point, assessing both its factual accuracy (true/false/undetermined) and its importance level for the given issue. Testing across six LLMs, the research team found that all models exhibited lower accuracy in verifying “direct-answer” statements compared to “supplementary statements,” a discrepancy that was obscured by conventional aggregate scoring metrics.
Further experiments show that when the validation task requires models to locate relevant information from longer texts or multiple sources, the scores of all models decline. For all four open-source models, incorporating both the original query and the LLM’s full response as context also leads to a significant deterioration in fact-checking performance. The paper emphasizes that these weaknesses could only be identified because JSFactCheckBench explicitly records the specific conditions underlying each annotation; if such metadata were omitted from the benchmark, these systematic biases would remain undetectable and unattributable.