We run a pipeline that pulls structured data out of about 1,000 customer emails a day — name, company, the product they mention, sentiment, and urgency. For the first three months it sort of worked, in the way that a car with a slow leak sort of works. Our JSON parse-error rate hovered around 12%. One in eight extractions came back broken: the model wrapped the JSON in a ```json fence, or opened with 'Sure! Here's the data:', or — my personal favorite — used single quotes because it was feeling stylistically adventurous. Every one of those was a dropped record or a trip through our increasingly grotesque JSON-repair function.
That repair function had grown to 120 lines. It stripped fences, fixed quotes, hunted for the first { and the last }, and re-tried parsing in four different ways. It was load-bearing duct tape, and I hated it. The real problem wasn't the parser — it was that I was asking the model nicely to return JSON and hoping. Hope is not a JSON strategy.
What the extraction prompt actually changed
I ran our extraction spec through the Craft a JSON-Only Data Extraction Prompt That Never Breaks Your Parser prompt and built the result on Claude Sonnet 4.6 — Sonnet is fast and cheap enough to run a thousand times a day and disciplined enough to follow an explicit output contract to the letter. The generated prompt opened with a line I'd never have written so bluntly: 'You output ONLY valid JSON and nothing else. You never write prose. You never use markdown fences. You never use single quotes. If you cannot extract the data, you return the JSON error object defined below.'
Then it defined the full schema — every field with a type, a nullable flag, and a description of exactly what to pull. The null strategy was the quiet hero: it told the model to use null for any missing optional field AND to include every key even when null. That single rule killed our 'KeyError: company not found' crashes downstream, because the shape of the object became guaranteed instead of hopeful.
The numbers after the swap
- Parse-error rate dropped from 12% to 0.08% across the first 10,000 calls on the new prompt
- The per-item confidence field (0.0-1.0) let us auto-route low-confidence extractions to human review instead of silently trusting them
- Our JSON-repair function went from 120 lines to 8 — it now only handles the one defined error-object format the prompt promises
- The handful of remaining 'errors' were genuinely un-extractable inputs (a two-word email, an auto-reply), and they returned the clean error object exactly as designed
Why I keep this one on the cheap, fast model
A thousand calls a day adds up, so model choice here is an economic decision as much as a quality one. I run this extractor on Claude Sonnet 4.6 specifically because it sits at the sweet spot: it's quick and inexpensive enough to call at volume without watching the bill climb, but it's disciplined enough to obey a strict output contract on the first try. I tested a heavier model and got marginally better extraction on the gnarliest inputs, but the cost and latency weren't worth it when the confidence field already routes the gnarly ones to a human. The right model for a high-volume extraction job is the cheapest one that holds the JSON contract — and Sonnet 4.6 holds it.
The step I now refuse to skip
The generated output included a TEST CASE — a real example input and the exact JSON it should produce — plus a tiny Python validation snippet. I used to deploy extraction prompts and find out in production whether they held. Now I run the test case first, confirm the snippet parses, and only then ship. It takes ninety seconds and it has caught two bad prompt edits before they hit the pipeline. One of those edits looked completely harmless — I'd reworded a field description — but it quietly changed the model's null behavior and would have started emitting empty strings where my downstream code expected null. The test case caught it in seconds. In production it would've been a slow, confusing data-quality bug I'd have chased for an afternoon.
The bigger mindset shift: I now treat an extraction prompt like a piece of code, because that's what it is. It has a contract (the schema), it has tests (the test case and validation snippet), and it has a failure mode (the defined error object). Treating it as 'just a prompt' I tweak and hope is exactly how I ended up with a 120-line repair function in the first place. Prompts that feed parsers deserve the same rigor as the parser.
If your extraction pipeline relies on a JSON-repair function you're quietly ashamed of, the fix is upstream. Grab the Craft a JSON-Only Data Extraction Prompt That Never Breaks Your Parser prompt on Prompt Dock and rebuild your extractor on Claude Sonnet 4.6. I feed the clean output straight into my Build an Inbound Email Triage Classifier with Confidence Scoring prompt — structured extraction plus structured classification is the whole backbone of our inbox automation, and neither half breaks the parser anymore.