A study submitted to arXiv on October 24, 2026 (paper ID 2610.31148v1) systematically examines the impact of tool augmentation on the comparative stability of language models. The research finds that, while equipping models with tools can improve accuracy on questions that programs can answer, it generally exacerbates fluctuations in model relative rankings caused by even minor changes in question phrasing.
The research team proposed a matching protocol that breaks down the tool‑calling process into five distinct stages: answer extraction, reasoning process, decision on whether to invoke the tool, the tool’s returned answer, and the default response when no tool is invoked. This protocol was applied to six open‑source models and tested across five knowledge domains and five different phrasing variants of questions. The results show that the extent to which formulation sensitivity expands is positively correlated with the frequency of model‑initiated tool calls, while the call frequency itself varies depending on the phrasing—this phenomenon is pronounced in some models but less evident in others.
Further analysis indicates that sensitivity is governed by interface and instruction design: in the molecular domain, after adopting native tool‑invoking interfaces and providing a second‑version tool description, no expansion of sensitivity was detected; however, in scenarios where tools return intermediate results, sensitivity persists. The study underscores that even when model outputs are deterministic, stable comparison outcomes across different formulations cannot be guaranteed, and recommends that future evaluations report both invocation frequency and comparative stability under multiple phrasing options, rather than relying solely on a single accuracy metric.