Getting structured output from an LLM that you can actually parse

Written By: BrainSoft In AI & Research

You ask the model for JSON, it hands back a friendly sentence wrapped around a code fence, and your json.loads throws at 2am. This is one of the most common integration problems we hit when wiring LLMs into existing systems, and it is almost never the model being dumb. It is the contract between your code and the model being vague.

The fix is to treat the model like any other untrusted service: define a strict schema, ask for it explicitly, validate every response against that schema, retry once with the validation error fed back, and keep a tolerant parser as a last resort. Below is the sequence we use on real projects.

1. Write the schema before you write the prompt. Pick your target shape as a real JSON Schema or a Pydantic model. The prompt is derived from it, not the other way around.
2. Use the provider's structured output mode if it has one. OpenAI's response_format with a JSON schema, or Anthropic tool use with an input schema, constrains decoding. It is not a guarantee, but it removes most of the noise.
3. State the contract in the prompt anyway. Even with constrained decoding, put the schema in the system message and say explicitly: no prose, no markdown fences, no trailing commentary.
4. Validate on arrival. Parse, then check types and required fields. Reject anything that fails before it reaches business logic.
5. Retry once with the error. Send the validation error back as the next user turn. One retry catches most transient formatting slips.
6. Fall back to a tolerant extractor. If the retry fails, strip fences, find the first balanced brace, and try again. Log the raw response when even that fails.

Here is the request shape we use with OpenAI's structured outputs. The schema is short on purpose; every extra optional field is another place the model can drift.

from openai import OpenAI
import json

client = OpenAI()

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "tags": {"type": "array", "items": {"type": "string"}},
        "confidence": {"type": "number"}
    },
    "required": ["title", "tags", "confidence"],
    "additionalProperties": False
}

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Return only JSON matching the schema. No prose."},
        {"role": "user", "content": article_text},
    ],
    response_format={"type": "json_schema", "json_schema": {"name": "summary", "schema": schema, "strict": True}},
)
data = json.loads(resp.choices[0].message.content)

That gets you most of the way. The remaining work is the validation layer, which is where the real reliability lives. We use Pydantic because it gives you typed objects and readable error messages in one step.

from pydantic import BaseModel, ValidationError

class Summary(BaseModel):
    title: str
    tags: list[str]
    confidence: float

try:
    parsed = Summary.model_validate_json(raw)
except ValidationError as e:
    # feed str(e) back to the model as a correction turn
    ...

Three details matter more than they look. First, additionalProperties: false and a full required list: optional fields invite the model to invent or omit. Second, keep field names boring and unambiguous. A field called date will sometimes come back as "next Tuesday"; call it iso_date and say so in the description. Third, if you need enums, declare them as enums in the schema rather than listing allowed values in prose.

The tolerant extractor is a small function worth writing once. It should handle a leading code fence, a trailing sentence, and a stray comma. If it still cannot parse, that is a bug report, not a silent fallback.

import json, re

def extract_json(text: str):
    text = re.sub(r"^```(?:json)?|```$", "", text.strip(), flags=re.MULTILINE)
    start = text.find("{")
    if start == -1:
        raise ValueError("no JSON object found")
    depth = 0
    for i, ch in enumerate(text[start:], start):
        depth += ch == "{"
        depth -= ch == "}"
        if depth == 0:
            return json.loads(text[start:i + 1])
    raise ValueError("unbalanced braces")

Log the raw response alongside the parsed result on every call, at least in staging. When something drifts, the raw text tells you whether the model ignored the schema or your parser did. We have seen both, and guessing wastes hours. If you are wiring this into a larger system, treat the LLM call as one adapter behind an interface, so replacing the provider later does not ripple through your codebase. That is the kind of thing we do regularly as part of our services.

Frequently asked questions

#AI & Research