Chunking: how retrieval systems cut a text apart

The term comes from retrieval systems, not from search engines. When a system has to answer with a specific quotation, storing the whole document is useless to it, because it would have to return thousands of words to support a single sentence. So it splits the document before storing it. Each resulting piece is a chunk, and that chunk is the smallest unit the system can retrieve and hand to the model.

Three things that tend to get mixed up belong apart. A paragraph is an editorial decision of yours. An HTML section is a layout decision. A chunk is a decision made by the system that ingests your document, taken with parameters you neither set nor see. Publishing three-line paragraphs obliges nobody to use those limits as chunk boundaries.

The second distinction concerns ownership, and it organises everything else. If you build an internal search or an assistant on your documentation, the chunking is yours: you choose the method, the size and the overlap, and you answer for the quality of the result. If the question is instead how to appear in a ChatGPT answer or in Google Search, the splitting happens inside somebody else's closed pipeline. The difference is not academic, because it determines what you can genuinely optimise and what you can only assume.

There are four families of method. The first cuts by fixed length, counted in tokens or characters, and is the simplest and the blindest, because the boundary falls where it falls. The second adds overlap: it repeats the end of one chunk at the start of the next so that a sentence cut in half still appears intact somewhere. The third cuts by structure, using headings, subsections or table rows as a natural boundary. The fourth cuts by semantic similarity, measuring where the topic shifts and splitting exactly there.

The concrete values depend on the system. Google's Vertex AI RAG engine starts at 1,024 tokens per chunk with 256 tokens of overlap, but that is one product's default and not a general recommendation. The documentation itself describes the trade-off without settling it: a small chunk produces more precise representations, and a large one produces more general representations that may lose details.

The characteristic failure appears when the context is left on the other side of the boundary. Anthropic illustrates it with a chunk stating that revenue grew by 3 % over the previous quarter, without naming the company or the period. Retrieved on its own, that piece is good for nothing, because it contains none of the words anyone would ask about. The remedy they proposed was to prepend to each chunk a sentence placing it inside its document before indexing. In their own evaluation this lowered retrieval failures from 5.7 % to 3.7 %; combined with lexical search it fell to 2.9 %, and with an added reranking stage to 1.9 %.

The decision hanging on this is a budget decision. If you run your own retrieval, chunk size and chunk boundaries are the cheapest lever you have on answer quality, ahead of switching models. It is worth measuring with your documents and your real questions before touching anything else.

If what you want is to be cited in search engines and assistants, the honest answer is less comfortable. Google documents a system called passage ranking, which identifies specific sections of a page in order to judge its relevance better, and presented it as an improvement affecting 7 % of queries. What that system does is documented. What boundaries it works with is not, and nowhere does Google state that the length of your paragraphs influences them.

For the advice derived from all this there is a direct source, and it says the opposite of the advice. Google states in its guidance on optimising for generative features that there is no requirement to break content into tiny pieces for AI to understand it, and that its systems handle multiple topics on one page. The budget line for rewriting an entire site into micro-paragraphs therefore has no documentary backing. What survives scrutiny is more modest and is good editing anyway: making each section understandable on its own.

• Establish first whose chunking it is. If it is yours, measure it with your data. If it belongs to a third party, treat it as a parameter you do not control, and distrust anyone selling you the opposite.
• Make every section stand on its own. Explicit subject at the start, with no «this», «the above» or «as we said» crossing a heading.
• Repeat the proper name instead of the pronoun when the referent sits in another section. It costs one word and saves the chunk.
• Close the figure together with its context in the same block: the number with its unit, its date and its scope, not spread across two sections.
• Use headings as a real boundary rather than typographic decoration, so that structure-based splitting lands where you want it to.
• If you run your own system, measure before copying. A set of thirty real questions with the correct answer recorded is worth more than any figure recommended in an article.

• Rewriting the site into two-line paragraphs in order to «get cited». Google says explicitly that content does not need to be broken into tiny pieces, so that effort is justified only if it improves readability.
• Copying a chunk size without knowing who measured it. The figures in circulation come from individual experiments, with a specific model and a specific task, and they do not transfer as they stand.
• Confusing passage ranking with chunking. The first is a ranking system documented by Google; the second is a preparation step in retrieval systems. They meet in the intuition and not in the mechanism.
• Leaving the overlap at zero and discovering afterwards that the important definitions fall exactly on the seam between two chunks.
• Splitting tables, lists and numbered steps down the middle. Half a table retrieved without its header is noise that looks like data.