On October 26, 2026, arXiv published a research paper titled “What the Query Remembers: Query Abstraction Governs Lexical and Dense Retrieval of Classical Arabic Poetry” (arXiv:2610.31821v1), which systematically examines how users’ residual memory of classical Arabic poetry affects information retrieval performance. The study compares nine dense encoders with the classic lexical retriever BM25 across two poetry corpora, covering three indexing methods and two evaluation protocols (known-item retrieval and topical relevance retrieval).
The results show that the relative performance of retrievers is not absolute, but rather determined by the original poetic‑line components embedded in the query. For “tip-of-the-tongue” queries—where users only vaguely recall some words—BM25 leads with a Recall@5 of 0.746, outperforming the best dense encoder at 0.717. However, under highly abstract queries, BM25 drops to eighth place among ten methods (0.392), far below the top dense encoder (0.748). Human‑edited experiments further confirm that simply adding a single line from the remembered half‑poem can boost BM25’s performance by 38 points, while manually rewriting the quoted phrases brings its scores level with those of the 8B dense model.
Mechanism analysis shows that BM25 performance is dominated by lexical inclusion, while dense encoders rely on proximity in embedding space; moreover, the two mainstream evaluation protocols exhibit significant divergence when applied to the same system—a phenomenon independently replicated by two blind human annotators and two open-source weight adjudicators (one of whom used a scoring criterion without literal-match requirements). The study emphasizes that reporting benchmarks based on a single protocol or a single query style can yield a severely distorted picture of system performance. All code and data have been made publicly available: https://github.com/Ellzo/what-the-query-remembers。