Structured outputs with strict schemas: catch refusals and truncated JSON before you parse

The response returns status 200, you are billed for every token, and yet the JSON inside does not match the schema you sent. With Claude, that is not a system fault but a refusal. Anthropic’s documentation is explicit: when the model refuses, the refusal message takes precedence over the schema constraints.

It is tempting to think that switching on strict is enough. In practice, strict only guarantees the JSON structure when the model answers in full. Two other cases remain: the model refuses, or the output is cut off midway because it hits the token limit. A Claude refusal looks exactly like a successful response. Truncated output does carry its own stop signal, but only those who read it will notice.

If you are an FDE putting a data-extraction pipeline into a customer’s system, these three branches are precisely where the demo and the production version part ways. The guide below builds each branch in turn, in the order you need to check them.

What will you build, and what do you need?

The running example is hypothetical. A function extract_invoice(text) reads a customer’s invoice emails and returns a dict with the vendor, invoice number, total and currency. When there is no usable result, the function raises a clearly named error rather than letting arbitrary exceptions escape.

You need Python, a Claude or OpenAI API key, and a few sample emails. The code blocks below have been simplified to make the logic easy to see. Check exact field names and SDK calls against the provider’s current documentation.

Step 1: which schemas will the API accept?

Start with the schema, because many 400 errors are already baked in here. Anthropic only allows additionalProperties to be set to false, and does not support recursive schemas, external $ref references, or numeric and string constraints.

Simon Willison, writing about OpenAI’s strict mode, adds that there every key of an object must appear in required. The schema below follows both rules so it works with either provider.

INVOICE_SCHEMA = {
    "type": "object",
    "properties": {
        "vendor": {"type": "string"},
        "invoice_number": {"type": "string"},
        "total": {"type": "number"},
        "currency": {"type": "string"},
    },
    "required": ["vendor", "invoice_number", "total", "currency"],
    "additionalProperties": False,
}

Note what has been deliberately left out: total has no minimum, and currency has no length limit. With Anthropic, using an unsupported feature returns a 400 error with details. On the Gemini side, Google also warns that schemas that are too large or too deeply nested may be rejected with 400 INVALID_ARGUMENT.

Check: send a test request. If you get a 400, read the error details and fix the schema before writing another line of code.

Step 2: send the request with the schema

With Claude, the schema goes in output_config.format with type: "json_schema", and the valid JSON sits in a text content block of the response. A simplified version:

request = {
    "model": MODEL,
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": prompt}],
    "output_config": {
        "format": {"type": "json_schema", "schema": INVOICE_SCHEMA}
    },
}

If you use OpenAI, do not confuse JSON mode with Structured Outputs. OpenAI states plainly that both produce valid JSON, but only Structured Outputs guarantees a schema match, and it recommends always using Structured Outputs. Willison’s view is that in 99% of cases you will want "strict": true switched on.

Check: measure latency from the second request onwards. Willison notes that the first request with each schema is slow because the schema has to be preprocessed. Benchmark on the first request and you will only alarm yourself.

Step 3: read the stop signal before parsing

This is the step most often skipped. Do not call json.loads straight away. With Claude, a refusal has a stop_reason of refusal, while truncated output has a stop_reason of max_tokens, in which case the JSON may be incomplete.

class Refused(Exception): pass
class Truncated(Exception): pass

def read_claude(response):
    if response.stop_reason == "refusal":
        raise Refused(response)
    if response.stop_reason == "max_tokens":
        raise Truncated(response)
    text = next(b.text for b in response.content if b.type == "text")
    return json.loads(text)

On OpenAI, the signals live elsewhere. Truncated output has a response.status of incomplete, with incomplete_details.reason set to max_output_tokens. Willison describes how, when the model refuses, you get back a separate refusal message rather than JSON following the schema. Because a refusal may not follow the schema, the API adds a refusal field that you must check before reading the parsed content.

# Simplified: the exact location of the refusal field and the content
# depends on the endpoint you use, check the OpenAI documentation
def read_openai(response, output):
    if response.status == "incomplete" and \
       response.incomplete_details.reason == "max_output_tokens":
        raise Truncated(response)
    if getattr(output, "refusal", None):
        raise Refused(output.refusal)
    return json.loads(output.text)

Situation	Claude	OpenAI
Model refuses	stop_reason is refusal, status 200, tokens still billed	A separate refusal field, not following the schema
Cut off by the token limit	stop_reason is max_tokens	status is incomplete, reason max_output_tokens
What to do before parsing	Read stop_reason	Read status, then the refusal field

Step 4: retry truncated output, but with a stopping point

Anthropic advises resending with a higher max_tokens to get the complete JSON. The rest is your design. One sensible approach is to double after each attempt, 1024 then 2048 then 4096, and stop after three tries so that one unusual email does not burn tokens indefinitely.

def extract_invoice(text, budgets=(1024, 2048, 4096)):
    for max_tokens in budgets:
        try:
            return read_claude(call(text, max_tokens))
        except Truncated:
            continue
    raise Truncated("token budget exhausted")

Refusals need entirely different handling: do not retry blindly. Every refusal is still billed for tokens, and resending the same input gives no reason to expect a different result. Push these cases into a queue for human review.

Check: call call(text, 50) with a long email. The Truncated branch must run, and json.loads must not raise a parse error.

Step 5: a valid schema does not mean valid values

Google says it bluntly in the Gemini documentation: syntactically correct JSON still needs its values validated in the application. The constraints you dropped from the schema in step 1 are rewritten here, in code that you control.

def validate(inv):
    errors = []
    if inv["total"] < 0:
        errors.append("negative total")
    if len(inv["currency"]) != 3:
        errors.append("currency is not a 3-letter code")
    if not inv["invoice_number"].strip():
        errors.append("missing invoice number")
    return errors

Picture an email that says only “total as per last week’s quote”. The schema forces every key because they are all in required, so you will very likely get back an empty invoice_number. The schema still matches perfectly, but the customer’s accounting system will not accept that row.

What mistakes are easy to make?

One easy mistake is parsing first and checking afterwards: a refusal then gets mistaken for a JSON error and logged under the wrong category. Another is leaving optional keys out of required when using OpenAI’s strict mode. And do not cram business rules into the schema, or you may lose a whole afternoon debugging 400 errors.

If you use strict tool use instead of an output format, remember that Anthropic limits each request to 20 strict tools. An agent whose tool list keeps growing over time will hit that limit sooner than you think.

How does this skill show up at the customer’s site?

Picture the first review on real data, when the customer asks what percentage of the pipeline works. The three named exceptions above give you the answer: count refusals, truncations and records that fail validation separately. Put these three numbers on a dashboard in the first week, before anyone asks.

For your career, a CV line such as “built an extraction pipeline using strict schemas, with separate handling for refusals and truncated output and per-category error metrics” shows recruiters you have shipped something to real production. When a job description mentions “LLM in production”, this is very likely the skill they need.

Getting schema-valid output is the easiest part of structured outputs. What makes you worth hiring is knowing what to do when the output is not valid.