LLM API costs are one of the few infrastructure expenses where usage and spend can diverge completely from user value. You can burn through your token budget on prompts that repeat the same system instructions a thousand times a day, on frontier-model calls for tasks a much cheaper model handles just as well, and on synchronous requests that could have been batched at a fraction of the cost. The good news is that most teams running LLMs in production are significantly overpaying — and the optimizations that close that gap are neither exotic nor risky.
First: Know Where Your Tokens Are Actually Going
You can't optimize what you haven't measured. Before implementing any cost reduction, spend a week logging every LLM call with its input token count, output token count, model used, latency, and the feature or use case that triggered it. Break this down by feature. The distribution is almost always surprising — one or two high-volume, low-value call patterns typically account for the majority of spend, while the complex reasoning tasks you built the system for represent a small fraction of token usage.
Most cloud providers expose per-request token counts in their API responses. Log them. Aggregate them. A simple dashboard showing daily token spend by feature, broken down into input and output, will reveal the optimization targets immediately.
Prompt Caching: The Fastest Win
If your system prompts, retrieved documents, or few-shot examples stay the same across many requests, you're paying to re-encode them from scratch on every call. Prompt caching lets you pay once to process a prefix and then reuse that cached computation for subsequent requests that share the same prefix.
Anthropic's prompt caching (available on Claude models) and OpenAI's similar mechanism both offer significant discounts — typically 50-90% — on cached input tokens. The practical steps:
- Structure your prompts so the stable prefix (system prompt, documents, examples) comes before the variable part (user query, current context). Caching only applies to the prefix.
- Keep the prefix as large as possible. A 10,000-token system prompt with supporting documents that's cached and reused across 500 daily requests saves vastly more than a 200-token system prompt.
- Measure cache hit rate. A cache hit rate below 70% suggests the prefix is varying in ways it shouldn't — debug why.
For a system with a fixed 8,000-token system prompt called 2,000 times per day, prompt caching can reduce input token costs by 60-80% on its own. This is usually the highest-ROI optimization available, and it requires almost no application logic changes.
Model Routing: Match Task Complexity to Model Tier
The second-biggest source of LLM overspend is using frontier models (GPT-4o, Claude Sonnet/Opus) for tasks that smaller, cheaper models handle just as well. A routing layer that classifies incoming requests by complexity and routes them to the appropriate model tier is one of the most impactful architectural patterns available.
| Task Type | Suitable Model Tier | Typical Cost Ratio vs Frontier |
|---|---|---|
| Intent classification, entity extraction | Small (Haiku, GPT-4o mini, Mistral 7B) | 3–10x cheaper |
| Short document summarization | Small to mid (Sonnet, GPT-4o mini) | 3–5x cheaper |
| Customer support Q&A with RAG | Mid (Sonnet-class) | 2–3x cheaper |
| Complex multi-step reasoning | Frontier (Opus, GPT-4o) | Baseline |
| Code generation (simple functions) | Mid (Sonnet, Codestral) | 2–4x cheaper |
The router itself can be a lightweight classifier — either a small LLM call that categorizes the request, or a rule-based system based on input length, feature flag, or query type. The classification call costs a fraction of a cent; the savings on routed requests pay for it many times over. Start simple: route only by input length or explicit feature type. Measure quality impact before adding complexity to the routing logic.
Output Length Control
Output tokens typically cost 2-3x more than input tokens. And LLMs will happily generate verbose, padded responses unless you constrain them. A few mechanisms:
- Explicit length instructions in the system prompt. "Respond in 2-3 sentences" or "Provide a JSON object, no prose" are dramatically more effective than you might expect. Models follow length constraints reliably.
- max_tokens parameter. Set it. Don't leave it at the model's default maximum. Know the typical output length for each use case and cap at 1.5x that value.
- Structured output formats. Asking for JSON or a specific schema reduces filler text and makes output easier to parse. It also makes output more consistent, which reduces retry rates.
A feature that was generating average 800-token responses — because the prompt said "be thorough" — often gets to the same user outcome with 250 tokens once the output expectations are made explicit. That's a 70% reduction in output token cost for that feature alone.
Batching for Async Workloads
Not every LLM call needs to return a result in milliseconds. Document processing pipelines, nightly data enrichment, bulk content generation, and scheduled report drafting are all workloads where latency is irrelevant and cost matters more than speed. Both Anthropic and OpenAI offer batch APIs that process requests asynchronously at roughly 50% of the standard per-token price.
The integration change is minimal: instead of calling the standard API endpoint, you submit a batch job, poll for completion (or receive a webhook), and retrieve results. For async workloads at volume, this is one of the easiest cost optimizations available. The only engineering cost is queuing and result retrieval logic, which is straightforward to build.
Semantic Deduplication and Response Caching
Some user queries are functionally identical even when the wording varies. "What's your return policy?" and "How do I return something I bought?" and "Can I get a refund?" all want the same answer. If you're running a high-volume customer support application, a semantic cache layer — that checks whether an incoming query is semantically similar to a recently answered one and returns the cached response — can reduce actual LLM calls significantly.
Implementation: embed the incoming query, check cosine similarity against a cache of recent (query, answer) pairs, and return the cached answer if similarity exceeds a threshold (typically 0.93-0.95). Set a short TTL (24-48 hours) so cached answers don't go stale. GPTCache and Redis with vector search are both reasonable infrastructure choices. The hit rate depends heavily on your traffic patterns — for support chatbots with a constrained question space, 30-50% cache hit rates are common.
A Realistic Optimization Roadmap
If you've never optimized your LLM spend, tackle these in order — the earlier items have the highest return on effort:
- Week 1: Instrument token usage by feature. Find the top 3 cost drivers.
- Week 2: Implement prompt caching on your highest-volume prompt patterns.
- Week 3: Audit output length instructions. Tighten them. Add max_tokens caps.
- Week 4: Identify which features are over-using frontier models. Test the same prompts against a smaller model. Route what passes quality bar.
- Month 2: Implement batch API for all async workloads. Add semantic caching for high-volume Q&A features.
Expect to see 40-70% cost reduction by the end of this roadmap. Some teams hit higher. The variance depends on how much waste existed in the original implementation, not on the difficulty of the optimizations.
Frequently Asked Questions
Does using a cheaper model mean lower quality for users?
Not necessarily, and often not at all. For many common tasks — extracting structured data from a form, classifying a support ticket, answering a factual question from a knowledge base — smaller models are within rounding error of frontier models in quality. The quality gap is real and meaningful for complex reasoning, nuanced writing, and multi-step planning. The most common mistake is applying frontier-model quality standards uniformly across tasks that don't require them. Measure quality on your actual tasks before assuming a quality penalty.
Is prompt caching supported by all LLM providers?
Not universally, but the major commercial providers have it. Anthropic supports prompt caching on Claude models with a clear prefix-based mechanism. OpenAI has automatic prompt caching for recent models. Google's Gemini API offers context caching with a minimum cache size. Open-source models deployed on vLLM or similar infrastructure support KV cache reuse, which achieves a similar effect. If you're using a provider that doesn't offer caching, that's a factor worth including in your provider evaluation.
What's the risk of model routing getting it wrong and sending a complex task to a cheap model?
It's a real risk, which is why you start with conservative routing — only route clearly simple tasks — and measure quality before expanding the routing rules. Build a shadow routing mode first: route everything to the frontier model for actual use, but log what the router would have done and compare the small-model outputs to the frontier outputs offline. This lets you set the quality bar before anything affects users.
How much can batching realistically save?
Batch API pricing on Anthropic and OpenAI is approximately 50% of standard pricing. For a workload that processes 100,000 documents per day and doesn't need real-time results — a nightly enrichment pipeline, a weekly content audit, a daily summary generation job — that's a straightforward halving of that workload's cost. Combined with prompt caching on the batch jobs, the effective discount can exceed 70% versus naive real-time API usage.
Need a partner for this? Mexilet offers generative AI development and AI solutions.
If your LLM costs are already material and climbing, this is worth a focused engineering sprint rather than a gradual side project. The team at Mexilet Technologies has helped companies audit and restructure their AI inference costs — typically as part of broader AI engineering work. If you'd like a free, no-obligation conversation about what the optimization potential looks like for your specific application, book a call with us here. Bring your usage numbers if you have them; even rough figures are enough to scope the opportunity.