Chunking strategies for production RAG

Everyone tunes the model and the prompt. Far fewer people tune the chunking — and yet it often decides more about answer quality than the model choice does.

Why chunk size matters

A chunk is the unit your retriever can return. Too large and you drown the model in irrelevant context; too small and you cut ideas in half so no single chunk is actually useful.

• Start around 500–800 tokens per chunk for prose.
• Add 10–15% overlap so ideas that straddle a boundary survive.
• Split on structure (headings, paragraphs) before splitting on length.

Retrieval quality is a data problem before it is a model problem.

The settings I ship

pythonfrom langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=700,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " "],
)
chunks = splitter.split_documents(docs)

This is a starting point, not a law. Measure retrieval hit-rate on your own questions and adjust — the right numbers depend on your documents.