On October 24, 2026, an arXiv preprint titled “Mostly Reading: What Open-Book Scores Measure in Document-Level LoRA Adapters” (ID 2610.31017v1) was published, raising critical questions about the current paradigm of knowledge editing and retrieval based on single-document LoRA adapters. The study trained dedicated LoRA adapters—referred to as “expert adapters”—for each of 150 Wikipedia articles using the Llama-3.1-8B-Instruct model, and designed two evaluation methods: open-book scoring (with explicit document text provided) and cloze probing (without source text), in order to decouple “reading behavior” from “knowledge retention.”
The results show that OpenBook scoring primarily reflects a general reading ability: a randomly initialized adapter trained on other documents can recover 82% of the OpenBook advantage that an expert adapter has over the base model; however, in a cloze‑completion task involving facts from the same document, this random adapter recovers only 2% of the advantage, and this value is statistically indistinguishable from zero. This indicates that, under the OpenBook setting, nearly all performance gains stem from the model’s immediate processing of the prompt text, rather than from domain‑specific knowledge encoded in the adapter weights.
Although the expert adapter does indeed outperform the base model by a statistically significant absolute improvement of +0.094 on the cloze‑filling task, confirming that content information is indeed encoded in the weights, adapter selection strategies based on weight‑space distance or text‑embedding similarity fail to demonstrate statistical superiority over random selection across both task types. Furthermore, when 16 or more adapters are combined, properly scaled fusion schemes can recover 87–89% of the open‑book advantage but only 12–16% of the cloze‑filling advantage (with confidence‑interval lower bounds below zero and upper bounds not exceeding 32%), whereas joint training achieves a 94% cloze‑filling advantage, further highlighting the limitations of decoupled adapters in knowledge retention.
The paper also specifically highlights four instances of “silent errors”: these did not trigger runtime exceptions, the output format was entirely correct, yet they led to incorrect conclusions—this detail underscores the fragility of method implementation and result validation in this field. All conclusions are based on the experiments presented in this preprint and have not yet undergone peer review.