On October 24, 2026, a research paper titled “Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering” was published on the arXiv platform (ID 2610.31135v1), systematically examining the differences in sensitivity to the position of answer-relevant evidence among Transformer, state-space models, and the hybrid Mamba architecture in long-context question answering.
The study quantifies how each architecture responds to changes in the position of evidence by sliding a gold-standard document bit by bit within a fixed context composed of nine perturbing documents. The results show that the evaluated Transformer models generally exhibit a “start‑of‑sequence advantage”—that is, they perform better when the answer‑relevant evidence is near the beginning of the prompt—whereas the evaluated state‑space models do not display this phenomenon; however, both types of architectures demonstrate a preference for evidence located near the end.
The experimental trends for pure Mamba and hybrid Mamba are consistent, but the statistical significance is insufficient; this family-level difference emerges only in models capable of handling basic question-answering tasks. In synthetic retrieval tasks, Transformers once again demonstrate an advantage, yet state-space models, having reached their performance ceiling, cannot support meaningful cross-family comparisons. Furthermore, switching the training corpus within the same state-space architecture alters overall accuracy without changing the shape of the position–performance curve, while prompt engineering can either introduce or virtually eliminate the initial advantage.
The study also found that, despite variations in response curves across different architectures, evidence‑location information can still be decoded at the cross‑architecture level; moreover, within the Transformer family, the degree to which attention is concentrated on the initial token covaries with its sensitivity to the input’s starting point. The authors emphasize that the effective utilization of long contexts depends on multiple factors, including architecture type, model capacity, the specific task, and prompt design—meaning that the sequence mixer alone is not the sole determining factor.