On October 24, 2026, a research paper titled “Delta N-gram: Forward-Only Continual Writing into a Language Model's Native N-gram Memory” was published on the arXiv platform (arXiv:2610.31125v1), introducing the Delta N-gram method, which aims to enable secure, efficient, and reversible continual writing to a language model’s native hash-based N-gram table.
This method strictly preserves the pre-trained table T⁰ unchanged, with all new knowledge written into the corresponding rows in the form of sparse patches Δ. A 4.1M-parameter writer module is trained during the offline phase according to the pre-training objective; at runtime, it requires only a forward pass, and clearing Δ fully restores the original model. On a 0.6B-scale prototype, Delta N-grams still maintain 16/16 facts after secondary writing, significantly outperforming online gradient-based writing, which achieves only 2–4/16.
The study compared two writers: the plain writer (which relies solely on T⁰) and the delta writer (which reads the current memory during encoding). Experiments on Qwen3.8-Flash-Next showed that the delta writer’s memory is more document-specific; after being exposed to 16 perturbing documents, its retention rate (70%) outperformed that of the plain writer (54%). Over 35,000 writes in a seven-task chain, the delta writer exhibited lower layer‑2 math drift (176% vs. 188%) and a smaller accuracy drop (-0.26 pp vs. -0.52 pp), with small‑k_i drift (17.9%/17.5%) significantly lower than gradient updates (39.2%). Furthermore, the research found that format‑related rows can trigger severe drift—rising to 4–12 times the pre-training norm—and that zeroing out just eight of these rows eliminates over 98% of layer‑2 math drift along with the corresponding NLL gain.