跳到正文
AI NEWS JOURNAL
arXiv cs.CLAI NEWS / JOURNAL

在真实文档修订下的块重新计算:衡量固定式、递归式与内容定义式分块的更新代价与检索质量

arXiv:2610.31019v1 发布类型:新 摘要:用于检索增强生成的分块策略几乎总是基于静态快照进行评估,然而知识库所依托的文档却在持续变化。由于各分块是独立嵌入的,编辑后移动的边界会导致未发生变化的文本也被重新嵌入。我们在严格的字节级复用规则以及冻结的稠密编码器条件下,针对三种无状态分块方法(非重叠的固定大小分块、递归的段落优先分块和内容自定义分块),对120对相邻的中文与英文维基百科修订进行了更新成本的量化分析。在包含116篇文章的共同成功测试集上,当参数被冻结且长度经过校准时,固定大小分块会重新计算其全部新版嵌入—输入工作量的57%至68%,递归分块……

对应语言译文排队中,完成后自动更新。

arXiv:2610.31019v1 Announce Type: new

Abstract: Chunking strategies for retrieval-augmented generation are almost always evaluated on a static snapshot, yet the documents underlying a knowledge base continue to change. Because chunks are embedded independently, a boundary that moves after an edit forces unchanged text to be re-embedded. We measure this update cost on 120 pairs of adjacent Chinese and English Wikipedia revisions for three stateless chunkers (non-overlapping fixed-size, recursive paragraph-first, and content-defined chunking) under a strict byte-identical reuse rule and a frozen dense encoder. On the 116-article common-success pool, with frozen, length-calibrated parameters, fixed-size chunking recomputes 57-68% of its full new-version embedding-input workload, recursive chunking 38-56%, and content-defined chunking 43-54% (it failed on 4 of 62 Chinese articles), while at these parameters mean evidence recall under a fixed 2048-token retrieval budget differs by less than two percentage points within either language. A cost-only parameter grid shows that the ranking of the two content-aware chunkers changes with chunk-length settings, whereas non-overlapping fixed-size chunking has the highest recomputation cost in every length-matched condition. Controlled edits and a verbatim-presence diagnostic support boundary drift as one contributor to recomputation of text already present in the previous version. Update cost is therefore a substantial, parameter-sensitive dimension that static retrieval evaluations do not expose and that should be reported alongside retrieval quality.

资料来源

arXiv cs.CL
← 继续阅读全部动态