Building a GenAI evaluation programme for a client: lessons from FDE work at Arize AI

The client’s AI team opens the demo of their customer-support assistant and adds one remark: “We’ve read a few dozen conversations by hand and they all look fine.” Next week they want to roll it out to every user. The question you need to answer now is no longer “is it good?” but “when it breaks, who finds out, and how?”

At Arize AI, that is almost the whole job. The company’s FDE job posting describes the role as working with clients’ AI teams to design, deploy and scale production-grade GenAI observability and evaluation programmes. FDEs also build custom integrations and tooling so the Arize platform fits each client’s requirements.

The skill transfers even if you never work at Arize. If you are putting LLMs into a client’s systems, be ready to answer that question with data rather than gut feel.

Observability and evaluation are two halves of one loop

Observability answers the question “what did the agent do?” Each request is recorded as a trace containing the prompt, the retrieved documents, the tool calls and the final answer. Phoenix, Arize’s open-source platform, builds its tracing on OpenTelemetry and is independent of vendor, language and framework.

That last detail matters to an FDE. The client may use any model or agent framework, and you only need to instrument against one common standard rather than rewrite for every stack.

Evaluation answers the question “did it do it right?” You turn traces into datasets, which Phoenix lets you manage by version for experimentation, evaluation and fine-tuning. In Arize AX, an experiment runs a task on every row of a dataset and scores it with an LLM judge, attaching an explanation to each score so you can fix the right kind of failure.

Example: a refund assistant and a “pretty good” judge

Imagine the client is a retailer with an agent that handles refund requests. You instrument the agent, collect a week of traces, pick 100 conversations and save them as the dataset refund-v1. Two of the client’s customer-support staff label them by hand: 80 pass, 20 fail.

Next you write an LLM judge and run it on the same 100 rows. The judge gets all 80 passing conversations right, but of the 20 failures it catches only 5. The results:

	Human: pass	Human: fail
Judge: pass	80	15
Judge: fail	0	5

Overall agreement is (80 + 5) / 100 = 85%, which looks fine on a slide. But recall on failures is only 5 / 20 = 25%: for every four broken conversations, the judge lets three through. Arize warns about exactly this situation: a judge with 85% agreement can still be unusable if it misses the failures you care about.

The code to compute both numbers is short, and you should always print them side by side:

def judge_report(human, judge):
    # human, judge: list of "pass" / "fail"
    n = len(human)
    agree = sum(h == j for h, j in zip(human, judge))
    fails = [j for h, j in zip(human, judge) if h == "fail"]
    caught = sum(j == "fail" for j in fails)
    return {
        "agreement": agree / n,
        "fail_recall": caught / len(fails) if fails else None,
    }

The fix starts with the judge’s explanations. Reading the 15 missed cases, you might find a common pattern: the judge only checks whether the answer is polite and consistent with policy. It has no idea the agent confirmed the refund before checking the order status.

A correct answer is not necessarily a correct process

The failure in that example lies not in the wording but in the path. The agent says “Your order has been refunded” yet skips the tool call that looks up the order. Arize advises that for agents you must evaluate both the final answer and the trajectory.

So version two of the judge needs to see the sequence of tool calls in the trace, along with a clear criterion: the order status must be checked before a refund is confirmed. Rerun the experiment on refund-v1 and compare recall on failures with the previous version. Because the dataset is held fixed, any difference comes from the judge, not the data.

Only once a judge has been validated against human labels does Arize recommend using it to gate releases, feed dashboards or drive automated routing. That order is a hard rule: building a gate on an unvalidated judge automates its blind spots along with it.

In what order should you do this at your client?

Start with traces, because without them every discussion of quality rests on gut feel. Next, work with the client to choose the failure types they genuinely fear, such as wrong refunds, data leaks or answers outside policy. That list determines which group you measure recall on.

Then build a versioned dataset and ask the client’s people to label it, ideally domain experts rather than engineers. After that, write the judge, measure agreement and recall, fix it using the explanations, and promote it to a release gate only when recall reaches a level the client accepts.

How you present results must also suit the audience. The Arize role calls for leading technical discussions with stakeholders ranging from engineers to senior executives. Engineers need the confusion matrix; executives need one sentence: “The judge currently catches 5 of 20 refund failures, so it cannot yet be used to block releases.”

The mistakes that kill evaluation programmes early

The most common mistake is reporting a single agreement figure, like the 85% above. The second is changing the dataset between two experiment runs, so nobody knows whether a higher score came from the prompt or from easier data. The third is scoring only the final answer of an agent that calls tools.

The fourth is about ways of working. FDEs at Arize run several engagements in parallel, from scoping to handover, and help write the team’s playbooks. If you build the process from scratch for every client you will not keep up, so turn the steps above into reusable templates.

How does this skill show up on a CV?

Arize’s posting asks for 2-5 years of software development experience, lower than many senior FDE roles. In exchange, candidates must have developed, deployed and continuously improved GenAI applications in production, and know cloud and container tooling.

Against a posting like that, a line such as “used LLM-as-a-judge” says little. A stronger line would read: “validated a judge against 100 human labels, raising recall on failures from 25% to a level sufficient to gate releases”. That tells an interviewer you know how to measure the right thing.

This week, take 50 traces from any LLM application you have, label them yourself, then let a judge score them. If the recall on failures surprises you, you have seen exactly what the client’s AI team has not.