On October 26, 2026, a research team from Tsinghua University and the Chinese University of Hong Kong open-sourced AKV—a KV cache engineering server built on the SGLang framework—designed to optimize the inference efficiency of large language model (LLM) agents. Unlike traditional methods that compress historical data through text summarization, AKV enables agents to dynamically discard certain KV cache entries and perform positional remapping (Repos) on the retained entries, all without modifying the original message sequence or requiring a full re‑prefill of the remaining text.
On the three model categories—GPT-OSS-120B, MiniMax-M2.7, and MiroThinker-1.7-Mini—and across the four agent benchmarks—BrowseComp-Plus, GAIA-Text, SWE-bench Verified, and xbench-DeepSearch-2510—AKV achieves an average inference cost reduction of 27.7% and a 1.7-point increase in average task scores. Specifically, for GPT-OSS-120B on SWE-bench Verified, the per‑task cost drops from 20.0 cents to 9.7 cents, while the score rises from 40.0 to 43.2. In high-concurrency (16) scenarios, P95 request latency is dramatically reduced from 229.90 seconds under the standard approach to 30.63 seconds, and the time required for full attention at each decoding step decreases from 7.082 milliseconds to 2.849 milliseconds.
AKV defines three core cache operations: Append (add new entries), Drop (discard specified entries), and Repos (use RoPE to rotate and reset key positions while keeping values unchanged). It extends SGLang’s radix tree structure to support drop/reposition nodes and, combined with a segment-batch attention mechanism, restores evicted states on demand. Its API is stateless and OpenAI-compatible, allowing users to declare policies via the drop_message and reposition fields. It also includes built-in round/token/model-driven strategies and provides CacheTool, enabling models to autonomously invoke drop_context.
Note: All performance metrics are based on the experimental results published in Table 1 of the paper and on the project’s official page. Costs are estimated as API‑equivalent values (with KV reads accounting for 82.4% of total API costs), and currently only Qwen, GPT‑OSS, and the MiniMax‑M2 series of text models are officially supported. Compared with other KV‑optimization approaches (such as StreamingLLM, SnapKV, etc.), AKV focuses on multi‑turn, tool‑calling‑oriented agent workloads; it does not rely on fine-tuning and does not claim compatibility with vision or multimodal models.