The Hidden Token Tax on JSON Schemas

Introduction 

Structured output has become one of the safest ways to integrate large language models (LLMs) into production systems. You pass a JSON schema to your LLM API call, and the response is guaranteed to conform to that structure. No parsing failures, no malformed responses, no retry logic. 

JSON schemas increase input token usage because the schema is included in every request sent to the model. Depending on the provider, this overhead ranges from almost negligible to more than ten times the original prompt size. 

In this article, you'll see measurements across multiple models and learn how to reduce that overhead. 

A quick primer for those less familiar with LLM billing: language models process text as tokens - fragments of words, roughly 3-4 characters each. You pay per token, both for what you send (input tokens) and what you get back (output tokens). When something silently adds tokens to your API call, it adds directly to your bill. 

The AI Center of Excellence (CoE) at Vonage adopted structured output across our pipelines - sentiment analysis, entity extraction, and customer interaction scoring. The improvement in developer experience was immediate. 

During testing, however, we noticed an unexpected pattern. Our token usage was consistently higher than what our prompts alone should have produced - not enough to trigger an alert, but enough to notice. We were running short analytical prompts, maybe 100 tokens each, through models with JSON schemas attached. The prompt size was trivial, so where were the extra tokens coming from? We set up an experiment to find out. 

Quantifying Schema Overhead  

The documentation for structured output tells you how to use it, but not what it costs. Every provider has a "Structured Output" page explaining the API parameters. None of them explain what happens to your token count when you attach a schema.  

We wanted to answer one simple question: when you pass a JSON schema to an LLM, how many extra input tokens does that actually add? 

The setup was deliberately minimal. Same prompt, same system instruction, same temperature. The only variable was whether a schema was attached, and how complex that schema was. 

We tested five schemas ranging from trivially simple to production-grade. The Minimal schema was just a single field (88 characters of JSON): 

{ 
  "type": "object", 
  "properties": { 
    "answer": { "type": "string" } 
  }, 
  "required": ["answer"] 
} 

From there, we scaled up in complexity: 

• Simple - three fields with an enum constraint (237 characters) 

• Medium - a nested object with arrays (358 characters) 

• Complex - arrays of objects with enums (555 characters) 

• Very Complex - deep nesting, multiple arrays, booleans, enums (1,730 characters) 

The prompt itself was a short customer service interaction consuming about 99 tokens on Gemini's tokenizer, 121 on Claude's. Different tokenizers split the same text differently, which accounts for the baseline gap between providers. We made the prompt deliberately small because we wanted to see the overhead in isolation, not buried under a long context. 

We ran it across three models: Gemini 2.5 Flash, Gemini 3.5 Flash, and Claude Haiku 4.5. 

How Much Overhead Does JSON Schema Add? 

The chart below shows the total input tokens reported by each model across all five schema complexity levels. The gray baseline bars represent the same prompt sent without any schema attached.

The results show a clear pattern. As expected, increasing complexity results in increasing token consumption across all three models. But there is one clear standout: Gemini 2.5 Flash stays close to the baseline throughout. For the deeply nested schema, Gemini 2.5 Flash uses 186 tokens compared to 1,189 from Claude Haiku 4.5. 

Token accounting differs slightly between providers, so these measurements reflect reported billing tokens rather than internal implementation details. 

Why Short Prompts Cost More  

Individually, tokens are cheap. But the cost picture changes depending on your use case. 

If you're running structured output against long documents - say, 5,000+ token inputs - the overhead is negligible. A few hundred extra tokens on top of 5,000 is a rounding error. 

But most production structured-output calls aren't long-context. They're short analytical tasks: 

• Classify this customer message (sentiment, intent, urgency) 

• Extract entities from this sentence 

• Score this interaction on five dimensions 

• Route this ticket to the right department 

These prompts are typically 50-500 tokens. At that scale, the overhead becomes the dominant contributor to your input token bill. 

Consider a medium-complexity schema on Claude Haiku 4.5 with a 100-token prompt: 

• Without schema: 121 input tokens 

• With schema: 415 input tokens 

• Overhead: 71% of your total input bill 

You're paying nearly 3.5x more in input tokens than your actual content warrants. Scale that across millions of API calls per month, and this becomes a meaningful cost driver. 

Choosing the Right Model for Structured Output 

Here's where it gets interesting for anyone evaluating models for production workloads. 

At a 100-token payload with a medium schema:  

• Prices as of July 2026 

Gemini 2.5 Flash is roughly nine times cheaper than Claude Haiku and eleven times cheaper than Gemini 3.5 Flash for this specific use case. Not because of base pricing alone, but because of how each model handles the schema internally. 

The base price ratios (Haiku is 2.7x more expensive than 2.5 Flash per token) get amplified by the overhead into much larger effective cost differences. 

When Overhead Stops Mattering  

The overhead is a fixed cost per call. As your input payload grows, it fades: 

• At 500 tokens (a short customer message with context) - the medium schema adds 37% on Haiku 

• At 2,000 tokens (a full conversation transcript) - down to 13% 

• At 5,000 tokens (a document summary or RAG context chunk) - under 6% 

• At 10,000 tokens (a long document or multi-turn history) - approximately 3% 

If your workload involves summarizing documents, running RAG over retrieved passages, or analyzing full conversation transcripts, the overhead is genuinely negligible. This is primarily a concern for high-volume, short-prompt workloads like classification, routing, and entity extraction. 

Why JSON Schemas Increase Token Usage  

When you pass a JSON schema via the API, models handle it in broadly two ways: 

Compact encoding - the schema is processed efficiently, possibly as special tokens or a compressed instruction. Gemini 2.5 Flash appears to encode schemas this way. Overhead is minimal and scales slowly with schema size. 

Verbose expansion - the schema gets converted into natural-language instructions that spell out every constraint explicitly. This is what the token counts suggest Gemini 3.5 Flash and Claude Haiku are doing, and it explains why their overhead scales roughly linearly with schema character count. 

The verbose approach likely produces more reliable schema adherence for complex structures, but it comes at a token cost that no provider documents. 

Reducing the Overhead 

Here's how to reduce it: 

• Keep schemas minimal. Remove fields you don't use in your downstream code. Flatten nested structures where the hierarchy isn't semantically necessary. Put descriptions and examples in your prompt, not in the schema definition itself. A schema shrunk from 1,730 characters to 555 saves you roughly 650 tokens per call on Claude. 
  On Claude, schema overhead ranges from 0.6 to 1.6 extra tokens per character of schema JSON, with smaller schemas carrying proportionally more overhead. On Gemini 3.5 Flash it's lower - roughly 0.3 to 0.5 tokens per character - but still adds up at volume. 

• Match model to workload shape. If your production workload is primarily short-prompt structured output (under 500 tokens of actual content), the overhead should be part of your model selection criteria. If your inputs are long (over 2,000 tokens), the overhead is negligible. Choose on quality and latency instead. 

• Consider skipping the schema for simple cases. For a single classification label or a boolean flag, a well-crafted prompt that asks for JSON directly costs zero overhead. You trade some reliability for cost savings - worth it when the structure is trivial. 

• Batch where possible. The overhead is a per-call fixed cost. Processing 10 items in one call pays it once instead of ten times. 

• Measure it. Run the same call with and without the schema. The difference in input tokens is your hidden cost. Multiply by call volume and per-token price. That number belongs in your cost model. 

Conclusion 

Schema overhead is a cost that doesn't show up as a separate line item in your bill. For teams running millions of short-prompt structured output calls, it can account for most of your input token spend. Measuring schema overhead and accounting for it during model selection is the first step toward reducing unnecessary AI costs. 

Methodology Note 

The experiments described in this article used identical prompts across all models and conditions: 

• Gemini models were tested via Vertex AI REST API (us-central1 for 2.5 Flash, global for 3.5 Flash) 

• Claude Haiku 4.5 was tested via AWS Bedrock 

• Temperature was set to 0.0 across all runs 

• Five schema complexity levels were tested, each run in both control (no schema) and with-schema modes 

• Token counts are as reported by each provider's usage API