FDE Times dataFDE jobs open 924▲33 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 80Companies hiring 196Remote-friendly 25%Median US pay $200kTop hirer Databricks 107
中文VI

The newspaper of the Forward Deployed Engineer

Guides

Testing a client's AI for bias by swapping who the applicant is

Dropping the gender column is not enough, but changing only demographic details in otherwise identical cases turns fairness into a bug you can measure.

ShareLinkedInFacebookX

In brief

  • Models learn from data that may already be skewed, so bias can appear even when nobody intends it.
  • Dropping sensitive columns may not be enough because proxy variables still carry the signal; measure directly with swapped pairs of profiles.
  • Prompt changes can reduce bias substantially, but for high-stakes decisions the AI should only advise while a human decides.
Chart of 200 dots, each representing one original case file. 31 dots shaded orange are the flipped files: variants that differ only in name and hometown received different verdicts. The hypothetical approval rate is 62% for a man from Hanoi and 51% for a woman from Soc Trang.
In the article's hypothetical swap test, 200 original files were turned into 800 calls, and 31 files had their verdict flipped solely by changing the name and hometown. Source: hypothetical results from the article, FDE Times.

It is your third week on site. The consumer-loan screening assistant you built is running smoothly. Then, in a review meeting, the head of compliance asks: “How can you be sure it doesn’t score men and women differently, or people from the north and the south?” You open the dashboard and find only latency, cost and accuracy.

None of those numbers answers the question.

Every FDE who puts an LLM into decisions that affect people will face it: hiring, credit, insurance, customer service. Engineers who answer with data and a process keep the client’s trust. Those who answer with a hunch will lose it sooner or later.

Bias does not need intent

Authors from McKinsey, writing in HBR, note that human bias is well documented, that it has begun to find its way into AI systems, and that companies deploying AI should treat reducing this risk as urgent.

The UK’s data protection regulator, the ICO, explains that AI learns from data that may be unbalanced or reflect past discrimination, so training on real-world data does not guarantee fair outputs.

NIST goes further: bias is present throughout technology processes and causes harm regardless of intent. For an FDE, that is why fairness cannot be treated as an ethics question to deal with later. It is a class of defect, and it should be tested like any other.

LLMs are not immune. Anthropic tested Claude 2.0 on high-stakes decision scenarios and found both negative and positive demographic discrimination in some settings. Note the word “positive”: a model that favours one group is also biased, and your test suite has to catch both directions.

Why removing the gender column is not enough

Many engineers’ first instinct is to strip gender, age and ethnicity from the prompt. The ICO says plainly that this may not achieve the goal, because proxy variables can reproduce the same old patterns.

Picture a loan application in Vietnam. The name “Nguyễn Thị Lan” already reveals gender (the middle name “Thị” marks a woman). A permanent address hints at region and even income. Graduation year reveals age. A self-introduction written in a regional dialect might lead the model to judge it “unprofessional”. You delete one column, but the signal survives intact in five others.

The counterfactual test, step by step

The most direct measurement is a counterfactual test: keep the entire content of a decision the same, swap only the demographic details, and see how the output changes.

If two applications are identical in income, debt and repayment history yet receive different verdicts only because one applicant is called Hùng and the other Lan, you have evidence.

Below is a minimal code skeleton for the loan-screening assistant. Each original profile is multiplied into several variants; the model returns “DUYỆT” (approve) or “XEM LẠI” (review).

import itertools, collections

VARIANTS = {
    "name_gender": [("Nguyen Van Hung", "male"), ("Nguyen Thi Lan", "female")],
    "region": ["Hanoi", "Soc Trang"],
}

def make_variants(base):
    for (name, gender), region in itertools.product(VARIANTS["name_gender"], VARIANTS["region"]):
        yield {**base, "name": name, "gender": gender, "hometown": region}

def run(profiles, call_llm):
    stats = collections.defaultdict(lambda: [0, 0])  # [APPROVED count, total]
    flips = 0
    for base in profiles:
        answers = {}
        for v in make_variants(base):
            out = call_llm(v)  # returns "APPROVED" or "REVIEW"
            key = (v["gender"], v["hometown"])
            stats[key][0] += out == "APPROVED"
            stats[key][1] += 1
            answers[key] = out
        flips += len(set(answers.values())) > 1
    return stats, flips

Two numbers matter. The first is the approval rate for each group. The second, and more important, is the number of “flipped” profiles: cases where variants of the same original profile receive different verdicts. Group rates can look balanced because biases in opposite directions cancel out; the flip count cannot hide them.

Suppose you run 200 original profiles with 4 variants each, or 800 calls. Hypothetical results: men from Hà Nội are approved 62% of the time, women from Sóc Trăng only 51%, and 31 of the 200 profiles flip.

An 11-percentage-point gap on financially identical applications is the number you take into the review, along with 5 concrete flipped examples for the client to read for themselves.

Fix it with prompts, but do not stop there

The good news is that Anthropic reports careful prompt engineering can significantly reduce both positive and negative discrimination. In practice you can add instructions telling the model to rely only on the financial fields, require it to give reasons criterion by criterion, or mask name and hometown before the input reaches the model.

After every change, rerun the same 800 calls and compare the flip count before and after.

But Anthropic itself states clearly that it does not endorse or permit the use of language models to make automated decisions in the high-stakes cases it studied. For a loan assistant, the safe answer is to design it so the AI only recommends and explains, while a human underwriter signs off on the final decision.

Put this into scope from the first week rather than waiting for an incident to raise it.

Good numbers are not the same as fairness

The ICO reminds us that statistical methods are only one piece of the puzzle and must sit within a broader, not purely technical, view of fairness. Zero flipped cases cannot tell you whether the scoring criteria were reasonable in the first place.

The ICO also notes that compliance with data protection law and compliance with equality law are separate matters: meeting one does not guarantee the other.

So the FDE’s job is to provide the technical evidence, while the conclusion that the system is “fair enough” belongs to the client and its legal and compliance teams. You prepare the numbers, examples and options; they decide the acceptable threshold.

Common mistakes

The most common mistake is testing only on a “happy path” set and concluding the system is unbiased. The second is measuring only group rates and ignoring the flip count, so opposing biases that cancel each other out go unseen.

The third is changing the prompt without rerunning the suite, or switching model versions and forgetting that the old fairness results no longer hold.

The hardest to spot is choosing the wrong variants. In Vietnam, region, ethnicity, forms of address and dialect often matter more than the axes English-language material tends to discuss. Ask the client which user groups they most fear will be disadvantaged, and put those groups into your variant set.

When interviewing for an FDE role tied to decisions that affect people, you can ask the employer how they test for bias; the answer tells you how much they value this skill.

On a CV, a line such as “built an 800-call counterfactual test suite and reduced flipped cases after prompt fixes” is more persuasive than any adjective about “responsible AI”.

Next time someone asks whether your assistant is fair, do not answer “probably”. Open the flip-count table.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
4 sources
Read next on the roadmap · Stage 6: MeasurementCatch prompt regressions in the pull request, before the client doesA GitHub Actions workflow runs promptfoo to compare old and new prompts on the same golden dataset, then posts the scores as a comment on every PR that edits a prompt.