Before building RAG, count tokens; then measure Recall@k

A client sends you a folder of operating procedures, warranty policies and a few FAQs, and asks for a chatbot that answers staff questions. Many engineers react the same way: they open the vector database docs, choose an embedding model and start debating chunk size.

Two weeks later the demo gets a very basic question wrong, and nobody on the team knows where the fault lies.

This happens a lot because people skip the two cheapest questions. Does the corpus actually need RAG? If it does, is the search step finding the right passage? For an FDE, answering both on day one can save weeks of work.

Before choosing an architecture, count tokens

In 2024, when it introduced Contextual Retrieval, Anthropic offered a very practical threshold. If a knowledge base is smaller than 200,000 tokens, roughly 500 pages, put the whole thing in the prompt; there is no need for RAG or anything like it. That was the 2024 figure, and the threshold is much higher today.

Prompt caching is what makes this workable. Anthropic says caching cuts latency by more than 2x and costs by up to 90%. Claude’s current documentation states that tokens read from cache cost 0.1 times the base input price, with a few model-specific exceptions.

If hundreds of staff ask question after question against the same set of documents, the most expensive part is paid only once.

According to the official documentation, newer Claude models have a 1M-token context window, equivalent to about 555,000 words. So don’t memorise the numbers. Learn how to check them again for the model and price list your client is actually using.

Putting everything in the prompt still needs measuring

Long context still carries a cost in quality; it just doesn’t show up on the invoice. The “Lost in the Middle” study (2023) found that performance can drop significantly when the position of relevant information in the prompt changes. A warranty clause buried in the middle of 400 pages can be missed, even though the model can “see” it.

So the real rule is to always measure, not “under 500 pages, you’re safe”. If you put the whole corpus in the prompt, you measure answer quality on a question set with known answers. With RAG, there is an extra layer to measure separately: retrieval.

Once you have RAG, the fault is usually in search

Hamel Husain’s evals-skills collection includes a very clear evaluate-rag guide: first establish whether the problem is in retrieval, generation or both, then fix retrieval first. The reasoning is simple. If the passage containing the answer never makes it into the prompt, no amount of prompt tuning will help.

Jason Liu is blunter still: always measure retrieval with precision and recall, especially recall, because missing a key passage is fatal. Husain’s guide also names Recall@k as the most important retrieval metric for a first pass. Recall@k answers one question: among the k results returned, is the correct passage there?

Anthropic reports its results in the same spirit. Rather than showing off better answers, it measures the retrieval failure rate in the top 20 chunks. Combining contextual embeddings with contextual BM25 cut that rate by 49%; adding reranking cut it by 67%. These figures measure the retrieval step alone, not the quality of the final answer.

Example: measuring Recall@k on warranty documents

Imagine the client’s corpus is too large for the context window, so you have split it into chunks. You don’t yet have logs of real questions. Jason Liu suggests a way to get started: ask an LLM to generate 5 questions each chunk can answer, then check whether the system retrieves that same chunk.

The data-generation prompt can be very short: “Here is a passage from a warranty document. Write 5 questions a customer service agent might ask that only this passage can answer.” Each question is stored with its source chunk_id. After that, the evaluation takes just a few lines:

def recall_at_k(eval_set, search, k):
    hits = 0
    for item in eval_set:
        results = search(item["question"], k=k)
        if item["chunk_id"] in [r.id for r in results]:
            hits += 1
    return hits / len(eval_set)

for k in (5, 10, 20):
    print(k, recall_at_k(eval_set, bm25_search, k))

Run it with k at 5, 10 and 20, because k is a parameter to measure, not guess. Anthropic found that passing the top 20 chunks to the model worked better than the top 10 or top 5. That finding concerns how many chunks to give the model for generating an answer, whereas Recall@k only measures whether the correct chunk is among the k results.

The two numbers answer different questions, and on a client’s data only the evaluation set can tell you which k makes sense.

Note the bm25_search line: start with full-text search. In 2024, testing on essays, Jason Liu found that full-text search and embeddings gave almost the same quality, but full-text search was about 10 times faster. A simple baseline gives you a reference point against which to compare every later improvement.

A process you can repeat for every client

Start by counting the tokens in the whole corpus and comparing them with the model’s context window and cache pricing. If it fits, put everything in the prompt, turn on caching and measure answer quality. If it doesn’t, build a question–chunk set, run a BM25 baseline and record Recall@k.

When retrieval is the bottleneck, Husain’s guide recommends optimising chunking with a grid search before adjusting generation. Try several chunk sizes and overlaps, and re-measure Recall@k for each configuration.

If recall is still not good enough, the logical next step is hybrid search and reranking, since those are exactly the two changes that let Anthropic cut retrieval failures by 49% and then 67%. Only once recall is solid should you move on to the prompt and the model.

The mistakes that send RAG projects off course

The most common mistake is looking only at the final answer. When the demo fails, the team edits the prompt, swaps the model and adds instructions, while the passage containing the answer was never put into the context. The second mistake is choosing k by feel, usually 5 because it “sounds tidy”.

The third is building RAG for a corpus of a few hundred pages or fewer, then carrying the extra weight of a chunking pipeline, an index and data syncing, when a cached prompt would have been enough. The opposite mistake is just as dangerous: believing a 1M-token window means there is no need to measure, and forgetting that the position of information still affects accuracy.

When reading FDE or AI engineer job descriptions, look for phrases such as “evaluation”, “retrieval quality” or “RAG in production”. On a CV, “built a RAG chatbot” says little. “Built a question–chunk evaluation set and raised Recall@20 from X to Y by changing chunking” shows you know how to measure.

Clients don’t care which vector database you use. They need to know that when a member of staff asks about a warranty clause, the system finds the right page, and a good FDE answers that with a measured number.