FDE Times dataFDE jobs open 8840 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 53Companies hiring 185Remote-friendly 25%Median US pay $200kTop hirer Databricks 98
中文VI

The newspaper of the Forward Deployed Engineer

Tools

OpenRouter swaps models in two lines, but evals decide if you should

Streaming usage chunks, price-based provider routing and fallback models raise no errors, yet each can skew results before anyone notices.

ShareLinkedInFacebookX

In brief

  • According to the official docs, changing base_url and api_key is enough for the OpenAI SDK to call OpenRouter. Switching models then means changing the slug.
  • The schema is very close to the OpenAI Chat API but not identical, and by default OpenRouter load-balances across providers with a preference for lower prices.
  • An eval suite that logs the model actually used is the evidence a client needs before trusting a model switch.
Diagram: an app calls OpenRouter. Two symmetrical branches lead from OpenRouter: branch ① switches provider to the cheapest, leading to the main model; branch ② (dashed) applies on error, leading to the fallback model. An orange bracket spans both models and points down to an orange bar: the eval log must record the model named in the response, otherwise eval scores will blend two models.
By default OpenRouter picks the cheapest provider, while fallback switches to a backup model when the main model fails. Neither raises an error, so every eval log line needs to record the model that actually answered, taken from the response. Source: OpenRouter documentation (quickstart, API reference, provider routing).

According to OpenRouter’s quickstart documentation, an application that already calls the OpenAI SDK needs only two changes to reach hundreds of other models: base_url and api_key. The docs describe this as a “drop-in” replacement.

For a forward deployed engineer, that ease is both useful and a little worrying. Sooner or later a client will ask whether they can try another model, perhaps a cheaper or faster one. The switch takes five minutes. Proving that the new model does not break the product takes an eval suite.

So an FDE needs to be able to do both: switch models cheaply through the OpenAI compatibility layer, and build an eval that shows whether the switch is worth making.

What do the two lines change?

OpenRouter, launched in early 2023, describes itself as a single endpoint for hundreds of models. The minimal example below uses exactly the two parameters the quickstart shows, except that the model slug is read from an environment variable:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

resp = client.chat.completions.create(
    model=os.environ["MODEL_SLUG"],
    messages=[{"role": "user", "content": "Summarize the following ticket: ..."}],
)

(The prompt reads “Summarise the following ticket: …”)

The docs say you can put any model slug in the model field. That is exactly why the slug belongs in a config file rather than in code. When the client wants to try a new model, you edit the config and run the eval, with no need to redeploy any logic.

“Drop-in” is not necessarily identical

OpenRouter’s API reference states plainly that its request and response schemas are very similar to the OpenAI Chat API, with a few small differences. The upside is that OpenRouter normalises every model and provider to one schema, so there is only one to learn.

One documented difference: when streaming, the final chunk carrying usage information has a non-empty choices array, contrary to OpenAI’s spec. Imagine a client’s code that identifies the usage chunk by checking for an empty choices.

After the move to OpenRouter, that branch may never run, and token counts go unrecorded until someone reconciles the invoice.

The lesson is that “it runs” does not mean “it runs correctly”. Before changing base_url in a client’s environment, review every place where the code parses responses itself, especially the streaming path.

Same slug, not necessarily the same provider

By default, OpenRouter load-balances across providers and prefers lower prices. The same model may therefore be served by different providers on different calls, and today’s eval results may not reproduce tomorrow.

There is another easy miss. The require_parameters option defaults to false, so a provider that does not support a parameter in your request can silently ignore it. Set it to true and OpenRouter will use only providers that support every parameter you send.

The fallback feature also needs to be understood properly. You pass an array of model IDs, and when the first model returns an error, OpenRouter automatically tries the next one, so a fallback model answers in place of the primary.

Users may then get an answer from a model you have not evaluated thoroughly, while billing follows the model actually used, whose name appears in the response.

An eval suite turns a hunch into evidence

OpenAI’s guide defines evals as testing model outputs against style and content criteria that you set yourself. The process has three steps: describe the task, run it on a set of test inputs, then analyse and iterate.

Imagine a client who wants to move support-ticket summarisation to a cheaper model. You assemble 50 real tickets with personal information removed, each paired with pass criteria: for example, the summary must state the correct order number and must not invent a refund promise.

Suppose the current model scores 46/50 and the new one 41/50. That five-ticket gap, together with the list of the five failed tickets, is what the client needs to see before deciding.

Every eval log line should also record the model that actually answered, taken from the response. If fallback kicked in partway through, that 41/50 is measuring a mix of two models, not one.

What to learn first

Start with the cheapest thing: write your own parser for streaming responses and check it against the differences listed in the API reference. Then move on to the harder skill of writing eval criteria.

When reading job descriptions, look for phrases such as “multi-model”, “LLM evaluation” and “model routing”. On your CV, rather than writing “experience with OpenRouter”, say that you moved an application to a different model through the OpenAI compatibility layer and backed the decision with a 50-case eval suite.

There is a simple test for your criteria. Take 10 outputs, have a colleague grade them using exactly the criteria you wrote, with no further explanation, and compare their grades with your own.

If the two of you disagree on many cases, the problem lies in the criteria, not yet in the model. Only when both graders reach the same result is a figure like 46/50 or 41/50 solid enough to put in front of a client.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
6 sources
Read next on the roadmap · Stage 5: DeploymentOn Redshift Serverless, the Cancel button doesn't cancel the billSnowflake, BigQuery and Redshift each meter something different, and only BigQuery can block an over-limit query before it costs the client anything.