A research paper titled “Compiling Autoregressive LLM Control into Policy-Exact Parallel Inference for Hierarchical RAG” was published on October 24, 2026, on arXiv (ID 2610.30664v1), proposing a new method to compile the autoregressive control logic of large language models (LLMs) in hierarchical retrieval-augmented generation (Hierarchical RAG) into policy-exact parallel inference.
This method abandons traditional serial syntax‑constraint generation (such as JSON output), instead constructing a shallow action prefix tree. It evaluates the distribution of actions under all prefix conditions through a single root prefill and a single tree‑causal probe, and leverages the “prefix‑state lemma” to ensure policy equivalence under exact arithmetic—yielding a control policy identical to that of serial decoding, yet requiring only two forward passes and zero output tokens. An additional probe pass can certify an accurate order‑marginalized MAP set.
The experiments were evaluated on the MetaX host device across four document‑exclusive datasets and two different scales: compilation control matched the sequential policy for 84–89% of cases, with all discrepancies arising from floating‑point computation approximations; control latency was reduced by 35–47% compared to token‑by‑token sequential decoding, and by 6–7% relative to an idealized chunking engine; replicate experiments on the H800 device confirmed that result ordering consistency was preserved.