Why swapping GPT for Claude takes more than a new endpoint

On 31 October 2026, OpenAI’s Evals platform becomes read-only for existing users. On 30 November 2026, it shuts down entirely. Any team that uses Evals to test before switching models has less than eight weeks to find a replacement.

A client asks whether Claude would do better, whether Gemini would be cheaper, or whether Llama could run on their own infrastructure. Imagine you are the FDE on that deployment and that question lands on your desk.

The easiest response is to swap the endpoint, keep the prompt and see what happens. But the official documentation from OpenAI, Anthropic, Google and Meta all suggests this does not work. A prompt that performs well has been tuned for one particular model. When you change models, the thing to protect is the test suite, not the wording of the prompt.

Prompts are written for a specific model

Start with OpenAI itself. The company recommends pinning production applications to a specific model snapshot, because even snapshots within the same family can behave differently. The risk exists before you have left OpenAI at all.

OpenAI also distinguishes between two styles of prompting. Reasoning models do better when given only a high-level goal. GPT models usually need precise, step-by-step instructions. A prompt written in detail so that GPT gets it right may be a poor fit for a reasoning model, and vice versa.

If the gap is that wide within a single vendor, a prompt is even less likely to survive a move to another one. The reason is that prompt writers tend to add lines that compensate for the old model’s weaknesses. Those lines stay in the prompt even when the new model no longer has those weaknesses.

A good GPT prompt can do harm on Claude

Imagine a prompt that extracts invoice data for a logistics company and has been tuned on GPT for several months. It uses markdown to divide its sections. It contains commands pushing the model to be thorough, such as “MUST read every line, MUST NOT skip anything”. At the end there is an extra step asking the model to check its own output. The assistant turn is prefilled with { to force the model to return JSON.

On a move to Claude 4.6, this prompt has four places that need review. The first breaks the integration immediately. Anthropic states plainly that, from the Claude 4.6 models onwards, prefilling the final assistant turn is no longer supported. The { trick for forcing JSON therefore has to be replaced with something else.

The second is harder to spot. According to Anthropic’s migration notes, Claude 4.6 is more proactive and may overreact to instructions that older models needed. The company advises toning down commands that push the model to be more thorough. Capitalised lines that once stopped the old model from missing things may now make the new one overdo it.

The third is the self-check step. Anthropic warns that re-check instructions carried over from prompts written for earlier models can cause the model to over-check, increasing token usage and latency.

The fourth concerns formatting and cost. An independent analysis in VentureBeat noted that OpenAI’s models work better with markdown, while Anthropic’s work better with XML tags for separating the parts of a prompt. The same piece pointed out that Anthropic’s tokenizer usually splits the same text into more tokens.

So the cost figures in the old spreadsheet no longer hold, even if the prompt is unchanged.

Gemini and Llama have different traps

If the destination is Gemini 3, the problem lies elsewhere. Google recommends direct, clearly structured prompts and leaving sampling parameters at their defaults. It warns that setting temperature below 1.0 can make the model behave unexpectedly.

If you habitually set temperature to 0 for extraction tasks, that habit is worth checking early when moving to Gemini 3.

Google also advises always including few-shot examples in prompts for Gemini. According to the company, prompts without examples are likely to be less effective. An instructions-only prompt that worked fine on another model will need a few sample input-output pairs.

Llama differs at a lower level. Meta’s models use their own prompt format with special tokens, and Meta’s documentation requires function calls to follow the specified format exactly. Tool support is also more limited. As a result, prompts and tool schemas written GPT-style cannot be copied straight across. The tool-calling layer almost always has to be rewritten.

The table below gathers these points into a checklist for each destination. The suggestions on XML tags and the tokenizer come from VentureBeat’s independent analysis, not from Anthropic’s official guidance:

Destination	Old habits that can cause failures	What to do
GPT (a different snapshot, or a reasoning model)	Not pinning a snapshot; step-by-step instructions for a reasoning model	Pin a snapshot; for reasoning models, switch to stating high-level goals
Claude 4.6	Prefilling the assistant turn; thoroughness commands; legacy self-check step; markdown sections	Drop prefill; tone down the commands; remove or shorten the check step (per Anthropic). Try XML tags and recalculate costs for the tokenizer (per VentureBeat’s analysis)
Gemini 3	Temperature 0; prompts without examples	Keep sampling at defaults; add few-shot examples; write direct, structured prompts
Llama	GPT-style tool schemas and function calling	Rewrite using the special-token format; make tool calls follow Meta’s format exactly

Write evals before fixing prompts

The table only tells you where to look. Whether the revised prompt is actually better can only be known by measuring. OpenAI recommends preparing representative fixtures, tests and eval checks before changing a production prompt.

Anthropic goes further and treats this as a prerequisite: before doing prompt engineering, you need clear success criteria and a way to verify them.

Automated tools do not replace this step either. OpenAI’s prompt optimizer warns that the prompts it rewrites can perform worse, and that they always need evaluation and manual review before going into production.

Go back to the invoice example. Suppose you have 50 real, labelled invoices, including blurry ones, multi-page ones and ones with missing fields. Run this set on the current model to get a baseline, say 46/50.

Then port the prompt to the new model, run it again and record three numbers: pass rate, average token count and average latency.

With that table in hand, the conversation with the client changes completely. Instead of saying “Claude seems fine”, you can say how many of the 50 cases the new model passes, which ones it fails, and how many more or fewer tokens it uses. Anthropic also notes that switching to a different model is sometimes easier than tuning the prompt.

But only with evals do you know which situation you are in.

If your team uses OpenAI’s Evals platform, the 30 November 2026 deadline is a reason to move your fixtures into your own repo now. Keep them as files, under version control, runnable against any model.

For destinations not covered in the table above, such as Grok, apply the same principle: read the vendor’s prompting documentation, list your old habits, and let the fixture set decide.

A fixture set is worth more than a line saying “proficient in prompting”

For a developer who wants to become an FDE, model migration is an excellent practice exercise because it is close to the real job. Take a prompt you have written, port it to two other models, build a small harness that runs a shared fixture set, then write a page recording what broke and how you fixed it. The result can go straight into a portfolio.

When reading job descriptions, look for phrases such as “eval”, “LLM evaluation”, “model migration” or “multi-model”. If you see them, move this experience to the top of your CV. Describe it in measurable results, for example “migrated an extraction pipeline from GPT to Claude, maintaining the pass rate on a 50-fixture set and reducing latency by removing a redundant self-check step”.

A line like that is more convincing than “proficient in prompt engineering”.

Models will keep being replaced, while a fixture set labelled from a client’s real data can be reused across every switch. If you have to choose what to do first, build the fixture set before optimising the prompt.