FDE at CoreWeave: a reference agent is for teaching, not for showing off

You arrive at the customer’s office on a Monday morning. Their AI team has an agent that runs nicely in a notebook but gives wrong answers when it meets real users, and nobody knows which step goes wrong. They do not need another polished demo. They need a way to look inside the agent and measure it.

That is exactly the work CoreWeave describes for its Forward Deployed Engineer, AI Agents role. The person in this role sits with the customer’s AI engineering team to design, prototype and ship agents to production using W&B Weave. On top of that, workshops, deep dives and enablement sessions are written directly into the job description, with the aim of helping customers adopt the product faster.

The two jobs are really one skill. The reference agent you build is the teaching material for the workshop, and the workshop is where that agent gets tested. This guide shows how to do both so that the customer’s team can carry on without you after you leave.

Why shouldn’t a reference agent be perfect?

Weave came to CoreWeave when the company completed its acquisition of Weights & Biases. CoreWeave’s documentation describes Weave as an observability and evaluation platform for tracking, evaluating and improving agents and LLM applications. The two words that matter are tracking and evaluating. A reference agent has to serve both, not just run nicely.

In Weave’s tracing model, each agent run is a conversation. A conversation is made of turns, and each turn holds the model call and its tool calls. For a teacher this structure is very useful: it lets you point at the exact turn and tool call where the agent went wrong, instead of saying vaguely that “the model hallucinated”.

So a good reference agent should deliberately contain one or two typical bugs that the customer’s team will meet in real life. If the agent gets every task right from the start, the traces have nothing to read and the eval has nothing to improve, and the workshop becomes a performance.

One example, end to end

Imagine the customer is a logistics company. Their AI team wants an agent that reads complaint emails, looks up the order and drafts a reply. Your reference agent needs only two tools: lookup_order(order_id) to look up the order and draft_reply(order, issue) to write the reply. You deliberately leave the prompt vague about how to pick out the order number when an email contains several numbers.

Next comes the task set. Weave’s agent-evals tutorial uses a judge model to score how well the agent completes each task, based on that task’s success criteria. You write 12 tasks from real emails that the customer has anonymised. Each task has a one-sentence criterion, for example: “The reply states the correct order number and expected delivery date, and does not promise a refund.”

An LLM judge alone is not enough. Weave scores with an LLM judge combined with custom scorers, and W&B’s agents page says plainly that scorers can be built-in, third-party or written yourself.

You add a very simple hand-written scorer: compare the order_id in the tool call with the correct answer. The judge rates the quality of the reply, while the scorer checks the facts without any judgement involved.

Suppose v1 finishes and the order_id scorer is right on only 7 of 12 tasks. In the workshop you open the trace of a failed task and walk through it turn by turn.

The model reads the email, calls lookup_order with a phone number instead of the order number, gets an error back, and still drafts a reply that sounds completely confident. The whole room can see the bug is in the first tool call, not in the writing step.

Now you hand the keyboard to the customer’s team. They fix the prompt and add a step that checks the order number’s format, producing v2. Weave lets you rerun the same task set against the new version and see whether v2 improved or regressed against v1 on the logged scores. Suppose the scorer rises to 11 of 12, but the judge scores one task lower than before.

That regressed task is the most valuable lesson of the session. The customer’s team sees for themselves that fixing one thing can break another. They also understand why the whole task set has to be rerun after every change, instead of retrying only the email that caused the bug.

How should the workshop be split?

A three-hour session can be divided as in the table below. The general rule is that the time the customer’s team spends at the keyboard should grow with each block.

Block	Duration	What the customer’s team does	What they take away
Reading traces	40 minutes	Trace the conversations, turns and tool calls of failed tasks	A list of failures by step
Writing tasks and scorers	60 minutes	Write more tasks from their own data, write one custom scorer	Their own eval suite
Fixing and comparing versions	60 minutes	Create v2, rerun, read improvements and regressions	A v1-versus-v2 comparison table
Agreeing next steps	20 minutes	Choose the first real agent to apply this to	A plan for the following week

The last block is the easiest to cut when the session overruns. Do not lose it. Without the name of a real agent and an owner, everything learned in the first three blocks will sit untouched in a notebook nobody opens again.

Common mistakes

The most common mistake is building the reference agent on your own data and stack instead of the customer’s. CoreWeave has committed to keeping the W&B platform interoperable, and for an FDE that commitment means you have to plug Weave into the framework and model the customer already uses. Do not make them switch to the tools you happen to know.

The second mistake is letting the LLM judge decide everything. A judge is useful for scoring wording, but facts that code can check, such as order numbers, amounts or dates, should have hand-written scorers. Otherwise the customer’s team will argue about the judge’s scores instead of fixing bugs.

The third mistake is treating the workshop as one-way delivery. The job description asks FDEs to gather feedback from the field and pass it back to shape Weave’s roadmap. Each time the customer’s team stumbles on a screen or has to write the same scorer again and again, note it down that same day, because it is data for the product team.

Turning this skill into an edge when applying

The listing on inferencejobs.com asks for Python, modern ML/AI frameworks and at least five years of software engineering or hands-on coding experience. Those are the baseline. But because workshops and enablement are written into the job description, evidence that you can teach others to build and measure agents is what you should put front and centre.

So add a reference-agent repo to your portfolio with all three pieces: a task set with criteria, at least one custom scorer, and a v1-versus-v2 comparison table that includes a regressed task.

Add the agenda of a workshop you have actually taught, even if only to colleagues. On your CV, write in terms of outcomes, for example “team X ran its own evals after the enablement session”, rather than “experienced with LLMs”.

When preparing for interviews, do not prepare only to talk about agent architecture. Prepare an enablement story too, and the strongest one is the time another team found a regressed task on its own without you pointing it out.