On October 25, 2026, researchers from Stanford University and the University of California, Berkeley publicly released TokenGate—a learnable token-pruning technique designed for the long-context pre-filling stage of large language models (LLMs). This module integrates into existing models in a “plug-and-play” manner, increasing the time-to-first-token (TTFT) by 1.4 to 11.3 times while keeping the original task accuracy unchanged, and reducing peak memory usage by 1.5 to 3.4 times.
The core mechanism of TokenGate involves inserting a small multilayer perceptron (MLP) gating module after a specific intermediate layer (denoted as layer L) in the Transformer, combined with lightweight fine-tuning using LoRA (r=16). This module computes an importance score for each token in the input document segment and then retains only the key tokens via Top-K selection; subsequently, the higher layers process only these K tokens and do not generate KV caches for them. Meanwhile, the prompt tokens are forcibly retained and cannot be pruned.
To address the issue that Top‑K selection is non‑differentiable, preventing end‑to‑end training, TokenGate introduces a MoE‑style router mechanism: it converts the scores of retained tokens into learnable bias terms on the attention logits, and further calibrates these using per‑head gains and a learnable temperature coefficient, enabling gradient backpropagation to the scoring module. Training employs the standard next‑token prediction loss, combined with both full‑context loss and random‑budget loss, ensuring that the gating module can adapt to various pruning budgets within a single training run.
The experiments were conducted using the Qwen3-1.7B and Qwen3.5-2B models, benchmarked across five datasets—RepoBench-P, 2WikiMultihopQA, MultiNews, BioDEX, and needle-in-a-haystack (NIAH)—with context lengths up to 128k tokens. The results show: on the NIAH-16k task, optimal performance was achieved, with TTFT speedup of 11.3× and memory reduction of 3.42×; at a 128k-context length, TTFT speedup reached 13.0×, while memory savings remained stable at approximately 3.43×; and on the 2WikiMultihopQA task, at the cost of 95% full-context accuracy, TTFT was accelerated by a factor of 3.0 and memory usage reduced by a factor of 2.7.
TokenGate also demonstrates robust performance in hybrid attention models (such as Qwen3.5-2B, which incorporates Gated DeltaNet and full‑attention layers): on the NIAH‑32k task, applying gating at layer 3 achieves full pre‑filling accuracy while running 3.8× faster than full pre‑filling and 2.1× faster than FastKV; if gating is applied at layer 1 or layer 2, acceleration can reach 7.8×, with an accuracy loss of only 6%. By contrast, baseline methods like FastKV can operate only on pure attention layers and cannot handle non‑standard structures such as DeltaNet.
This technology has been open-sourced, supporting Python 3.10+ and CUDA PyTorch 2.9+, with the optional integration of the FlashAttention-2 key‑bias kernel for further acceleration. The GitHub repository provides Qwen3‑1.7B checkpoints for all five tasks, as well as a dedicated gated model for the NIAH task based on Qwen3.5‑2B (e.g., Hugging Face model ID: liana313/tokengate-niah-qwen3-1.7b). Training takes approximately 1.3 times as long as standard LoRA fine-tuning, but four out of five tasks converge more quickly, with an average training time of 1.8 A100‑hour‑equivalents (compared to 2.2 hours for the baseline).