RAG Is Not Dead: When Retrieval Still Beats a Bigger Context Window

Every time a frontier lab ships a bigger context window, the same claim resurfaces: RAG is dead, just stuff everything into the prompt. It's wrong often enough that we still build retrieval pipelines for most production clients — here's the actual decision boundary.

What a Bigger Context Window Actually Fixes

Larger windows genuinely help when your knowledge base is small enough to fit entirely and static enough not to need frequent updates — a product's documentation, a fixed set of policy documents, a single contract under review. In those cases, retrieval adds complexity and a failure point (bad retrieval = wrong context) without a corresponding benefit.

Where RAG Still Wins, Clearly

• Knowledge bases too large to fit in any context window economically — retrieval scales to millions of documents, brute-force context stuffing does not
• Frequently updated data — a retrieval index updates in minutes; re-processing everything through a giant context window on every query doesn't scale operationally or cost-wise
• Cost-sensitive, high-volume production use — even a model with a huge context window charges more tokens for a huge context every single call, while retrieval sends only the relevant chunks
• Source attribution requirements — retrieval naturally tells you which document an answer came from; a model reasoning over a giant blob of stuffed context is much harder to make auditable

The Real Answer: Most Production Systems Need Both

The best-architected systems we build use retrieval to narrow a large corpus down to the genuinely relevant subset, then rely on a generous (but not maximal) context window to reason over that subset with room for conversation history and tool outputs. Treating this as an either/or choice is the actual mistake — not picking RAG or large-context specifically.