FDE Times dataFDE jobs open 924▲33 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 80Companies hiring 196Remote-friendly 25%Median US pay $200kTop hirer Databricks 107
中文VI

The newspaper of the Forward Deployed Engineer

Guides

Teach your RAG agent to search again instead of bluffing

A grade-and-rewrite loop catches near-miss passages, and 20 real questions with known answers show whether the loop actually helps.

ShareLinkedInFacebookX

In brief

  • Basic RAG embeds the question and matches it against a vector database exactly once. Agentic RAG turns retrieval into a tool that the agent decides whether to call.
  • A grader that returns structured output decides which branch comes next. If the documents are not relevant, the system must not answer from them; it must rewrite the question instead.
  • In a client deployment, the loop needs a cap on retries, a way out such as "not enough evidence", and answers that cite their sources.
Flow diagram of an agentic RAG loop: the original question enters a step that decides whether documents need to be looked up; if not, the agent answers directly. If so, it searches the vector store, then a grader (highlighted in orange) scores relevance as yes or no. Yes: answer with sources [id]. No: check whether rewrites remain; if so, rewrite the query using the grader's reason and return to the search step; if none remain, decline because of insufficient evidence. Notes: the grader scores against the original question, max_rewrites=2, and the grader uses structured output.
A search–grade–rewrite loop: the grader scores evidence against the original question. When it returns “no”, the agent rewrites the query and searches again. Once the rewrites run out (max_rewrites=2), the system declines to answer rather than guess.

LangGraph’s agentic RAG tutorial contains a rule that sounds obvious: if the grader judges the retrieved documents irrelevant, the graph must not answer from that context. It rewrites the question and searches again instead.

The rule deserves attention because it blocks one of the most predictable ways RAG makes things up. The model receives a few passages that are nearly right and tries to answer with them anyway. For a Forward Deployed Engineer, this is exactly the kind of error to catch before the client does.

This guide moves from a simple RAG pipeline to a loop of deciding, grading and rewriting. All the code is a simplified sketch in plain Python. retrieve(), llm() and llm_structured() are functions you wire up to your own vector store and model; they are not part of any library’s API.

What will you build, and what do you need?

Take a hypothetical example: a chatbot that answers questions about a company’s HR policies. An employee asks: “Are probationary employees entitled to annual leave?” Vector search returns three passages about leave for permanent staff. They are semantically very close to the question, but they do not answer it.

You need three things: a vector store loaded with documents, a model that can return JSON against a schema, and about 20 real questions with correct answers. The question set is the most valuable of the three, because without it you have no way of knowing whether the loop helps.

Step 1: Build a one-shot retrieval baseline

AWS describes basic RAG this way: the question is converted into a vector and matched against a vector database. The matching happens once.

def simple_rag(question):
    docs = retrieve(question)   # embed the question, match once
    return llm(f"Answer based on:\n{docs}\n\nQuestion: {question}")

Check: run the 20 questions and record how many are answered correctly. On the probation question, the baseline will quite likely answer confidently using the rules for permanent staff. Keep this number for comparison at the end.

The LangGraph tutorial builds a retrieval agent that decides for itself when to search the vector store and when to answer directly. Retrieval becomes a tool call rather than a fixed step.

def decide(question):
    v = llm_structured(
        prompt=f"Does this question require looking up internal documents? {question}",
        schema={"need_search": "yes|no"},
    )
    return v["need_search"] == "yes"

Check: pleasantries such as “thank you” should go straight to the answer branch, and every policy question should go to the search branch. If the agent skips the search on a policy question, fix the prompt before going further.

Step 3: Write a grader that returns structured output

This is the core of the design. In the tutorial, LangGraph grades documents with a routing function, and that function uses a model with a structured output schema. The reason is practical: code needs to read a clear label; it cannot parse a sentence like “seems fairly relevant”.

GRADE_SCHEMA = {"relevant": "yes|no", "reason": "str"}

def grade(question, docs):
    return llm_structured(
        prompt=f"Can the following passages answer the question?\n{docs}\n\nQuestion: {question}",
        schema=GRADE_SCHEMA,
    )

The reason field is a design choice in this sketch: it tells the rewrite step where the previous search went wrong.

Grading evidence also has academic grounding. The Corrective RAG (CRAG) paper adds a lightweight evaluator that scores the overall quality of the documents retrieved for a question, then uses the confidence it returns to choose the next retrieval action.

Check: on the probation example, the grader should return "no" with a reason along the lines of “the documents only cover permanent staff”. If the grader labels everything "yes", your grading prompt is too lenient.

Step 4: Route on state

LangGraph uses conditional edges: the next node is chosen at runtime by running a function on the current state. The plain-Python version below does the same thing.

def route(state):
    if state["grade"]["relevant"] == "yes":
        return "generate"
    if state["rewrites"] >= state["max_rewrites"]:
        return "give_up"
    return "rewrite"

The give_up branch is another design choice in this sketch. Without it, a question the documents cannot answer would keep the system looping forever.

Step 5: Rewrite the query and close the loop

The rewrite function takes the grader’s reason for failure:

def rewrite(question, reason):
    return llm(f"Rewrite the question for better search. "
               f"The previous attempt failed because: {reason}\nQuestion: {question}")

The main function decides whether to search, then initialises the state:

def agentic_rag(question, max_rewrites=2):
    if not decide(question):
        return llm(question)
    s = {"query": question, "rewrites": 0, "max_rewrites": max_rewrites}
    return loop(question, s)

The loop retrieves, grades and routes:

def loop(question, s):
    while True:
        s["docs"] = retrieve(s["query"])
        s["grade"] = grade(question, s["docs"])  # grade against the original question
        nxt = route(s)
        if nxt == "generate":
            return answer_with_sources(question, s["docs"])
        if nxt == "give_up":
            return "Not enough evidence found in the documents to answer."
        s["query"] = rewrite(s["query"], s["grade"]["reason"])
        s["rewrites"] += 1

Note one detail: the grader always scores against the original question, never against the rewritten one. The rewrite is used only for searching. This stops the loop from drifting towards a different, easier question.

Check: print the sequence of queries for the probation question. The second attempt should produce something like “leave policy for probationary employees”, and this time the grader should return "yes".

Step 6: Answer with sources

AWS lists source attribution as one of the benefits of RAG. Once evidence is being graded, citing sources also lets users check the grader for themselves.

def answer_with_sources(question, docs):
    ctx = "\n".join(f"[{d['id']}] {d['text']}" for d in docs)
    return llm(f"Use only the passages below, citing [id] after each point.\n{ctx}\n\nQuestion: {question}")

Check: rerun the 20 questions and compare against the step 1 baseline. Beyond the number of correct answers, count two more things: questions the system refused that it should have answered, and questions it answered that it should have refused.

What are the most common mistakes?

The first is letting the grader return free text and guessing the label with a regex. The system works in the demo, then breaks on an answer like “yes, but not completely”. The second is an unbounded loop. For questions outside the document set, model costs climb with nothing to show for it.

The third is harder to spot: the rewrite keeps the keywords but loses a condition. The word “probationary” disappears, for instance, and the question reverts to a generic one. To catch this, read the trace of each rewrite rather than only the final answer.

Self-RAG takes a different route: it trains the model itself to retrieve only when needed and to critique its own output using special reflection tokens. The loop in this guide needs no training, which makes it easier to deploy at a client: any model that supports structured output can run it.

How does this skill show up at a client?

NVIDIA sees the future of RAG as LLMs and knowledge bases orchestrated dynamically into autonomous assistants. Even so, be ready for client conversations that start not with architecture but with a specific complaint, such as “the bot gave the wrong answer about policy X”.

The first thing to do is switch on tracing, then show the client where the grader said “no” and what the system did next.

On your CV, do not just write “built a RAG chatbot”. Write that you added evidence grading and query rewriting, and state how much the number of wrong answers fell on a set of real user questions.

An interviewer may ask how you handled questions that the documents could not answer. The give_up branch and its trace make a better answer than any slide.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
5 sources
Read next on the roadmap · Stage 3: Applied AIStop asking the big model what to do after every stepSplitting a loan-case agent into planner, DAG check, executor, critic and replan makes it measurable, though a fixed process needs only a hand-written workflow.