The eval harness that grades six contract memos with no API key
The repository behind the book Forward Deployed AI Engineering shows how to measure whether an AI pilot is good enough to go into production.
In brief
- The ghanemzadeh/ai-fde repo accompanies the book Forward Deployed AI Engineering and contains templates, prompts, an eval harness and illustrations.
- The harness is written in Python, has 36 tests and runs offline; its dry run grades six fictional contract memos and flags the ones that fail.
- The harness code is MIT-licensed; the templates, prompts and illustrations are copyrighted and may be adapted but not republished as a set.
Six contract review memos, all fictional, are graded against a rubric. At the end of the run, the tool reports which ones failed and why. That is the dry run in ghanemzadeh/ai-fde, the repository that accompanies the book Forward Deployed AI Engineering, and it runs without an API key.
The repo holds templates, prompts, an eval harness, illustrations and reference material for the book, which runs to five parts and twenty chapters, with twelve templates. The README is blunt about its purpose: everything in it exists to be copied and adapted.
For an FDE, the harness is the part to open first. Gyde describes the role as revolving around the gap between an AI pilot and AI running in production. HatchWorks argues that the FDE owns the whole loop, from the question of what to build to the point where the system is live and has real users.
In the middle of that loop sits a question every client eventually asks: how do you know the output is good enough? A pilot that cannot answer it with numbers usually stays a pilot.
Four components, each catching a different kind of failure
The harness is a Python project with 36 tests. It combines deterministic checks, a model acting as judge, a step that calibrates the judge’s scores against human graders, and a dashboard. The easiest way to understand it is as a stack of filters.
Deterministic checks are cheap and certain: is a required section missing, is the format right? The model judge handles what a regex cannot catch, such as whether the reasoning is actually grounded in the contract’s clauses.
Calibration is the component most often skipped. A model judge is still a model, and it can be lenient or biased in ways nobody anticipated.
Say the client’s lawyer regrades all six memos and agrees with the judge on only four. Those two disagreements show where the rubric or the judge’s prompt needs fixing, before anyone presents the dashboard to the board.
Because the harness runs offline, the whole grading flow can be read without spending anything on API calls. That makes it a better way to learn evals than a slide deck: run it, see which memo fails, then trace back to the code that failed it.
How much can safely be reused?
The licensing comes in two parts. The harness code is MIT-licensed and can go into client projects. The templates, prompts and illustrations are copyrighted: they may be adapted for your own work but not republished as a collection. For a portfolio, publish a harness you have transformed, not a copy of the template set.
The templates are worth reading alongside the code, because an FDE’s job does not end at the engineering. HatchWorks describes the FDE as part consultant, part product manager, part senior engineer, and the single person accountable.
Developers looking to move into FDE work have a clear exercise ready. Fork the harness, replace the six contract memos with output from a domain you know, write your own deterministic checks, and hand-grade a small sample to calibrate the judge.
On a CV, a line such as “built an eval harness with judge calibration against human graders, reporting agreement rate” says more than any claim of “LLM experience”. When reading job postings, look for companies that mention evals or measuring output quality: they are likely the places where this skill is valued directly.
When a client asks how you know the system answers correctly, do not reply with a hunch. The most convincing answer is a run that lists exactly where it is going wrong.
Was this article useful?
Thanks for the feedback!