Thomson Reuters hires FDEs for CoCounsel: every level evaluates LLMs, and Senior needs a legal background

In its ad for a Senior Software Engineer on the CoCounsel Forward Deployed Engineering team, posted on Dice, Thomson Reuters lists an unusual requirement for a software engineering role: candidates need a JD, formal legal training, or equivalent deep experience in a legal practice setting. The company is hiring a software engineer, yet it is asking about law school.

There is a reason. The same ad tasks the hire with evaluating non-deterministic AI output for legal accuracy, relevance and reliability. To judge whether a legal answer is right or wrong, you have to know how a lawyer would read it.

As FDE work moves deeper into specific industries, being able to call a large model’s API is no longer enough to stand out. You also have to prove the output is correct by the standards of people in that field, which means writing evals, and that work weighs as much as building the pipeline.

Pipelines remain the core job, with evals from the first level

The CoCounsel FDE team has posted ads at two levels. The mid-level ad lists building and operating RAG pipelines, agentic workflows and multi-component AI systems among its responsibilities. The Senior ad on Built In NYC, taken down on 20 August 2026, also covers building RAG pipelines and agents.

Directly beneath the building work, though, sits evaluation. Read the ads side by side and it is clear that evals are not reserved for senior staff.

Aspect	Mid-level	Senior (Built In NYC)	Senior Legal (Dice)
Building	RAG, agentic workflows, multi-component AI systems	RAG pipelines and agents	—
Role in evals	Helps evaluate LLM output	Leads LLM evaluation	Evaluates non-deterministic AI output
Grading criteria	Legal accuracy, relevance, citations, consistency, hallucination	Output quality, hallucination risk	Legal accuracy, relevance, reliability
Specific requirement	—	—	JD, formal legal training or equivalent experience

At mid-level, evals already appear, with the most detailed list of criteria of the three ads. At Senior, the verb changes from “help” to “lead”, alongside benchmarking output quality and measuring hallucination risk.

In other words, the more senior the role, the more evals shift from something you do to something you own. The pipeline does not disappear at any level.

Why is grading legal AI so hard?

Thomson Reuters explained this itself in a 2024 article on benchmarking CoCounsel. Traditional benchmarks rely on multiple-choice or true/false questions, and according to the company, that approach is not enough to evaluate the open-ended tasks lawyers face every day.

Harder still, “correct” sometimes comes down to preference. The article notes that two answers can both be right and differ only in what the user prefers, which makes it difficult to automate evals consistently.

In 2026, introducing CoCoBench, Thomson Reuters settled on its yardstick: every output is graded by whether a practising lawyer would consider it acceptable. The company also noted that systems optimised for general benchmarks often struggle when graded on real workflows.

Consider an example. You build a contract review assistant for a law firm and test it on 20 questions. Eighteen answers are substantively right, so if you grade only “right or wrong”, the system scores 90%.

But when you check each citation, you find that 3 of the “right” answers cite clauses that do not exist in the contract. Counting citations, only 15 answers are both right and correctly sourced: 75%.

A lawyer will not look at the 90% figure. They will see that roughly one answer in seven invents its source, and they will stop trusting the tool.

The example shows why it makes sense for the mid-level ad to list citation quality and hallucination risk as separate criteria alongside legal accuracy: lumped together, fabricated sources hide easily behind a flattering number.

The example has one more layer. Thomson Reuters stresses that if initial retrieval is poor, LLM performance drops with it. So with those 3 fabricated citations, the first step is to check whether retrieval pulled back the right clauses, before blaming the model.

The lesson applies to any client. Before tuning prompts or switching models, agree with domain experts on which errors are unacceptable, then turn each error type into its own metric, measured separately for retrieval and for the answer.

Is there still a chance without a law degree?

The legal-background requirement is in the Senior ad, and the wording includes an “or”: a JD, formal legal training, or equivalent deep experience in a legal practice setting. The door is not fully closed to engineers who never studied law, but you have to show you understand how lawyers work.

The reason lies in CoCoBench’s own yardstick. If the grading standard is what a practising lawyer would accept, whoever writes the evals must be able to tell an answer that “reads fine” from one a lawyer would put their name to. No pipeline can fix an eval suite that grades wrongly from the start.

The same logic applies to every vertical, from healthcare to finance.

The eval suite is what lets you switch models

There is another, practical reason evals matter. According to a November 2024 press release, Thomson Reuters’ LLM evaluation programme tests new models from multiple AI providers directly inside its products.

A multi-model strategy only works with a stable eval suite to compare against. Each time a new model arrives, the question “should we switch?” can only be answered by rerunning the same questions against the same criteria and putting the numbers side by side.

On the client side, FDEs face the same question, usually when nobody has prepared for it. If you build the eval suite in the first week, the model-switch meeting becomes a session spent reading a table of numbers rather than an argument based on gut feeling.

One portfolio project that tells the whole story

If you want to prepare for a role like this, you can start with a single project. Build a RAG system on a public corpus of legal documents, such as template contracts or statutes, then add a simple agentic step, for example automatically breaking a complex question into several smaller ones.

The most important part is the accompanying eval suite. Each question needs a reference answer and a source clause. Score retrieval accuracy, answer accuracy, citation accuracy and the fabricated-source rate separately, and record how the numbers change with each iteration.

These metrics track the criteria listed in the mid-level ad closely. If you can, run the same eval suite on two different models to add a comparison table.

On your CV, do not just write “built a legal chatbot with LLMs”. Write along the lines of: designed a citation eval suite, found how many fabricated-source errors and brought them down to what level.

In interviews, describe a time you sat down with domain users to decide which errors were unacceptable. That story shows you understand what Thomson Reuters uses as its yardstick: what practitioners accept, not what general benchmarks score highly.

The CoCounsel team’s ads show that a vertical FDE is an engineer who can both build the system and measure correctness by the standards of people in the profession. Anyone who can do both will have an edge in every industry where a single wrong citation can cost a client.