Temperature 0, different results: why one eval run is not enough to decide

Picture this: it is Thursday, and the client asks what looks like an easy question. Is the new prompt better than the old one? You run your 200-question eval suite. The old prompt scores 84%, the new one 80%, and you are about to write “rollback” in the report. Then you rerun the old prompt with exactly the same configuration, temperature 0, and this time it scores only 81%.

The bug is not in your code. Thinking Machines states plainly that, in practice, LLM APIs are still not deterministic, even at temperature 0. For an FDE, that means any “keep or drop” decision based on a single eval run may be nothing more than noise.

One prompt, 1,000 runs, 80 answers

The Thinking Machines experiment is simple: send the same prompt to Qwen3-235B 1,000 times at temperature 0. The result was 80 distinct completions, and the most common one appeared only 78 times. The outputs were identical for the first 102 tokens, then diverged from token 103.

Do the arithmetic. 78 out of 1,000 is 7.8%. If you took one “canonical” output as the reference answer and used exact matching, more than 92% of runs would fail to match, even though nothing changed on your side.

You might assume that an eval suite asking only short-answer questions is safer. Not necessarily. A study on arXiv (2408.04667) tested 5 LLMs configured to be deterministic on 8 common tasks, with 10 runs per configuration, in both zero-shot and few-shot settings.

The result: accuracy varied by up to 15% between ordinary runs with no intervention, and no model delivered consistently repeatable accuracy. What drifts here is the accuracy figure you were going to put in your report, not just the wording of the output.

Why doesn’t temperature 0 save you?

The common explanation is that “GPUs run in parallel, floating-point additions happen in no fixed order, so results drift.” Simon Willison, summarising the Thinking Machines piece, stresses that this hypothesis is not the main cause.

The real culprit sits outside your request. Server load determines the batch size the kernels run with, load changes unpredictably, and the computed results depend on batch size. Thinking Machines calls this a lack of batch invariance, rather than requests leaking information into one another.

From there it is easy to see what happens to tokens: a tiny numerical deviation is enough for two tokens with nearly equal probability to swap places, and one different token mid-sentence pulls the rest of the answer in another direction. That is exactly the picture of 80 completions sharing the first 102 tokens and then splitting apart.

The root-cause fix is batch-invariant kernels; the code released with the Thinking Machines piece makes Qwen3-8B run deterministically on vLLM. But that can only be done on the infrastructure side.

If the client hosts its own models, this is a question worth putting to the inference team. If you call an external API, treat nondeterminism as a given and design your evals around it.

Seed and fingerprint: useful, but only “best effort”

Many engineers think setting a seed settles the matter. The OpenAI Cookbook chooses its words carefully: the system will make a “best effort” to sample deterministically. The Azure OpenAI documentation is more explicit: even when the seed and system_fingerprint match across calls, it is not uncommon to see different responses.

system_fingerprint is still worth having. According to the OpenAI Cookbook, it represents the backend configuration the model is running on, and should be used together with the seed to detect when the backend changes in ways that may affect determinism.

Azure adds two further warnings relevant to evals. By default, asking the same question several times is likely to return different answers, so reproducibility requires keeping both the seed and parameters such as temperature and max_tokens fixed. And the larger max_tokens is, the less deterministic the response.

Indeed Engineering goes a step further: the variance of the model’s own output can exceed the variance from sampling the eval data. In other words, noise from the model can be larger than the noise from which 200 questions you happened to pick.

Example: building a repeated-run harness

Back to Thursday. Instead of running each prompt once, you run each prompt five times and record everything. The code below uses the OpenAI Python SDK; grade is your grading function, returning 1 if correct and 0 if wrong.

import statistics
from collections import Counter

def run_eval(client, cases, model, n_runs=5):
    scores, fingerprints = [], Counter()
    for _ in range(n_runs):
        correct = 0
        for case in cases:
            r = client.chat.completions.create(
                model=model,
                temperature=0,
                seed=42,
                messages=case["messages"],
                max_tokens=case.get("max_tokens", 256),
            )
            # record the fingerprint to see whether the backend changed mid-run
            fingerprints[r.system_fingerprint] += 1
            correct += case["grade"](r.choices[0].message.content)
        scores.append(correct / len(cases))
    return {
        "mean": statistics.mean(scores),
        "min": min(scores),
        "max": max(scores),
        "fingerprints": dict(fingerprints),
    }

Imagine the results: the old prompt ranges from 79% to 85%, the new one from 78% to 84%. The two ranges almost entirely overlap. The honest answer to the client is “no difference detected yet”, not “rollback”.

Now imagine the opposite: the new prompt ranges from 88% to 91%, so its lowest score is still above the old prompt’s highest. Then you have evidence.

Why not just run once to save time? Another arXiv study (2509.24086) shows that leaderboards based on a single run are fragile: 10 of 12 slices flipped at least one pairwise ranking compared with the majority result of three runs. The authors recommend at least 2 repetitions when using stochastic decoding.

Comparing min-max ranges over five runs is still a rough rule. For more confidence, Indeed Engineering notes that multiple outputs per input produce clustered data, so when computing confidence intervals, bootstrap by input: resample each question together with all of its runs, rather than pooling every output into one.

And if the fingerprints column shows two different values within the same batch of runs, split the results by fingerprint before drawing conclusions, because you may be comparing two backends rather than two prompts.

Try it on your own project

Start by measuring baseline noise: run your current eval suite five times with an unchanged configuration. The min-max range you get is your “noise floor”; any difference smaller than that should not be presented as an improvement.

Next, log system_fingerprint, model, seed and max_tokens for every call. When the client asks why this week’s score differs from last week’s, you will have data to answer with instead of guesses.

Then review how you grade. With long outputs, exact string matching will almost certainly break; ask the model to return a structured field and grade on that field, or grade against criteria rather than word by word.

The most common mistakes

The most common mistake is comparing two single runs and drawing a conclusion. With variation of up to 15%, as measured in arXiv 2408.04667, a 4-point gap from one run tells you nothing.

Next is believing that seed plus temperature 0 guarantees reproducibility. The provider itself promises only “best effort”, and the Azure documentation says plainly that output can still differ.

The hardest to spot is writing snapshot-style regression tests for long outputs. The tests will fail at random, the client’s team will learn to ignore them, and when a real bug arrives nobody will notice.

A skill worth putting on your CV

If an interviewer asks “how do you know the new prompt is better?”, don’t stop at “the eval score was higher”. Answer with a process: repeated runs, case-by-case comparison, the same fingerprint, a reported range. That answer shows you understand your data.

On a CV, a line such as “built an eval harness that reports variance across multiple runs and tracks backend changes via system_fingerprint” is far more concrete than “experienced with LLM evaluation”. When a job description mentions evals or reproducibility, bring exactly this example into your cover letter and interview.

The client doesn’t need a pretty number. They need to know whether that number will still hold next Thursday.