# Two experts, two verdicts: measure Cohen's kappa before you trust an eval set

> Two people agreeing 80% of the time sounds fine, until you subtract the overlap due to chance and are left with a kappa of 0.375.

Original: https://fdetimes.net/en/guides/cohens-kappa-eval-label-agreement/

In your third week at a customer, you have 100 chatbot answers labelled pass or fail, and the automated grader is tracking those labels fairly closely. Then you ask a second expert to grade the same 100 answers. On 20 of them, the verdict comes back the opposite way.

The first reflex is to ask who is right. The better first question is whether these two people are grading the same thing. If they are not, every score from the eval set merely reflects how the original labeller thinks, and you are about to report those scores as a measure of quality.

Cohen's kappa answers that question in a few minutes and a few lines of code. For an FDE, it is the cheapest check available before building an entire measurement system on an answer key nobody has reviewed.

## Why does 80% agreement say so little?

Raw agreement is easy to calculate, but as McHugh notes in her 2012 review, its key weakness is that it ignores the possibility that raters guessed. When the data is skewed, that weakness becomes serious.

Arize offers a tidy example: 100 samples contain 5 errors, one rater finds all 5, and the other passes everything. They still agree on 95% of samples, even though the second rater caught no errors at all. Calculate kappa for this exact case and the result is 0.

The formula has only two quantities: κ = (po − pe) / (1 − pe). Here po is the observed agreement rate, and pe is the agreement expected by chance, estimated from how often each rater uses each label. The numerator is the margin above chance; the denominator is the largest margin possible.

## A case worked from start to finish

Imagine you are evaluating a claims chatbot for an insurance company. The claims team lead (A) and a compliance officer (B) independently grade the same 100 answers, using a one-line rubric: "Is the answer correct?" The results look like this.

| | B: pass | B: fail |
|---|---|---|
| **A: pass** | 70 | 10 |
| **A: fail** | 10 | 10 |

The two agree on 70 + 10 = 80 answers, so po = 0.80. Each passes 80% and fails 20%, so pe = 0.80 × 0.80 + 0.20 × 0.20 = 0.68. In other words, two people grading independently at these rates could already have matched on 68% of answers.

So κ = (0.80 − 0.68) / (1 − 0.68) = 0.12 / 0.32 = 0.375. Take out the overlap due to chance, and an agreement rate of 80%, which sounds very healthy, leaves a kappa of just 0.375.

The function below also handles multiple labels because, according to Arize, Cohen's kappa is not limited to pass/fail and works for two raters across several classes.

```python
from collections import Counter

def cohen_kappa(a, b):
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    count_a, count_b = Counter(a), Counter(b)
    labels = set(a) | set(b)
    pe = sum(count_a[k] * count_b[k] for k in labels) / n**2
    return po, pe, (po - pe) / (1 - pe)

a = ["pass"] * 70 + ["pass"] * 10 + ["fail"] * 10 + ["fail"] * 10
b = ["pass"] * 70 + ["fail"] * 10 + ["pass"] * 10 + ["fail"] * 10
print(cohen_kappa(a, b))  # po≈0.80, pe≈0.68, kappa≈0.375
```

If both raters give every sample the same single label, pe equals 1 and the function divides by zero. The problem then lies in the sample: it is not varied enough to measure anything.

## Is 0.375 good enough?

There is no universal threshold. The Landis and Koch scale, which classes 0.61–0.80 as substantial and 0.81–1 as almost perfect, is widely cited but not universally accepted. Fleiss divides the range differently: above 0.75 is excellent, 0.40–0.75 is fair to good, and below 0.40 is poor.

McHugh goes further, objecting that Cohen's original scale, which treats only κ ≤ 0 as no agreement, is too lenient for health research. She regards any kappa below 0.60 as inadequate, and says results built on such labels merit little confidence.

The practical advice is to settle the threshold with the customer before calculating, and to record which scale it comes from. For a chatbot that touches claims money, 0.60 is a reasonable floor to start from. A score of 0.375 falls below both the Fleiss and McHugh thresholds, so this set of labels should not yet serve as ground truth.

## Fix the rubric; do not look for a winner

Now comes the most valuable part: sitting down with both experts and reading the 20 disagreements. Suppose a pattern recurs: the answer states the right payout but omits the exclusion conditions. The claims lead passes it because the information is correct; the compliance officer fails it because the customer could misread it.

Neither is wrong. The rubric never said what counts as "correct". You add a rule with an example: omitting an exclusion condition that affects the customer's entitlements is a fail. That rule is itself a product finding the customer will want to hear.

**Key point:** A label two experts cannot agree on has not earned its place as ground truth.

Suppose round two, on 100 new answers, produces 72 shared passes, 18 shared fails, 6 answers where A passes and B fails, and 4 the other way round. Now po = 0.90 and pe = 0.78 × 0.76 + 0.22 × 0.24 = 0.6456, so κ ≈ 0.72.

That clears McHugh's 0.60 threshold and counts as substantial on the Landis–Koch scale, yet is still only "fair to good" for Fleiss. One kappa, three descriptions: this is why a report should give the number first and the adjective second.

## When errors are rare, do not read kappa alone

Kappa depends on prevalence (whether the labels occur evenly) and bias (whether the two raters use the labels at different rates). With rare classes, it tends to understate true agreement. Once a chatbot is running well and fails have become sparse, this is exactly the situation you will face.

Take 100 answers: 94 shared passes, 2 shared fails, and 2 disagreements in each direction. Raw agreement is 96%, pe = 0.9232, and κ ≈ 0.48. Turn just two disagreements into agreements, giving 95 shared passes and 3 shared fails, and κ jumps to about 0.74 while pe stays the same.

Two answers are not enough to conclude that a rubric is broken or sound. When you hit this case, open the disagreement table, then filter the logs for more suspected fails and have both raters grade them blind.

A sample filtered this way has a higher fail rate than the original, and because kappa depends on prevalence, the two numbers cannot be compared directly. Report both, along with the fail rate of each sample.

## Clean labels first, then the LLM judge

Human-to-human agreement also serves as the reference point for assessing the automated grader. Arize advises comparing the LLM judge with each expert pair by pair, and treating the figure between the two humans as a benchmark rather than a ceiling.

Do not merge the two raters' labels into one before comparing. Suppose the judge reaches a kappa of 0.70 with A but only 0.40 with B: you know at once whose thinking it leans towards, which a merged label would hide.

## A process to take to the customer

1. **Draw a sample.** Take a random sample from real logs, not the cases the customer picked to show off.

2. **Grade independently.** Bring in the two people who will actually answer for quality, give them the same rubric and ask them to grade separately, without conferring.

3. **Calculate and read.** Calculate kappa, check how often each rater uses each label, then go through each disagreement with them.

4. **Fix the rubric.** Turn each group of disagreements into a rule with an example.

5. **Grade a new sample.** Do not reuse the old one, because regrading only measures how well they remember the answers.

6. **Lock it in and record it.** Only when kappa clears the agreed threshold should the labels become ground truth and be used to validate the judge.

Record the sample size, fail rate and rubric version, so that whoever takes over later knows where the number came from.

## Mistakes that make the number meaningless

- **Letting experts confer while grading.** Kappa goes up, but it now measures the persuasiveness of whoever talks most, not the clarity of the rubric.

- **Reporting only percentage agreement on skewed data.** The 95% example above shows that figure can hide a rater who catches no errors at all.

- **Treating a scale as law.** A number landing in a flattering box is not enough to declare a pass.

- **Removing the "outlier" rater.** That person may be holding on to a criterion the customer needs.

## How do you tell this story in an interview?

When applying for FDE roles, show that you are willing to doubt your own yardstick. If the job description mentions evals or ground-truth datasets, prepare a specific case for the interview: where the labels diverged, how you discovered it, and how the rubric changed.

On a CV, a line such as "raised kappa between two experts from X to Y over two rounds of rubric revisions, before deploying an LLM judge" says more than a whole paragraph listing tools. It shows you know how to measure and how to work with the people who own the domain.

This week's exercise: take 50 samples from an eval set you currently trust, ask a colleague to grade them blind against the existing rubric, then compare with the old labels. The result may be uncomfortable, but better to learn it early from a colleague than to hear it from a customer at the acceptance meeting.

**Try this week:**

- Copy the cohen_kappa function from this article, run it on the 70/10/10/10 table, then switch to 95/1/1/3 to see how kappa reacts.
- Add a line recording kappa, sample size, fail rate and rubric version to the document describing the eval set you use.
- Reread an FDE job description that mentions evals and write two sentences on how you check the reliability of labels.

## Sources

- [Cohen's kappa – Wikipedia](https://en.wikipedia.org/wiki/Cohen%27s_kappa)

- [Interrater reliability: the kappa statistic (McHugh, Biochem Med)](https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/)

- [How to measure human-LLM judge alignment (Arize)](https://arize.com/blog/measuring-human-llm-judge-alignment/)