AI Cost Optimizer

The three LLM cost levers — ranked by effort vs. impact

Prompt Cache is the highest-ROI, lowest-effort change: one line of code to fix your system prompt position, and input costs drop 90% for cached tokens. Batch API requires minimal code changes for async workloads and saves 50% across all models. Model switching has the highest absolute saving potential but requires quality evaluation — budget time for evals before deploying in production.

Why LLM costs compound faster than expected

Unlike fixed infrastructure, LLM costs scale with every user interaction. A product with 1,000 DAU each making 5 API calls per day is 150,000 requests/month. At GPT-5 pricing without optimisation, that can easily reach $15,000–$50,000/month. The same workload with Prompt Cache, Batch API, and a model switch to GPT-5 Mini can cost under $2,000/month.

The three levers of LLM cost optimization

Every dollar you spend on LLM inference passes through three independent levers, each capable of cutting costs 30–50% on its own. The first lever is model selection: right-sizing your model for each task instead of routing everything through the most expensive option. A classification task that works fine on GPT-5 Nano at $0.05/$0.40 per million tokens does not need GPT-5 at $1.25/$10.00. The second lever is input optimization: techniques like prompt caching, shorter prompts, context compression, and removing redundant few-shot examples that reduce the number of input tokens billed per request. The third lever is request optimization: batching multiple requests into a single Batch API call, deduplicating identical or near-identical queries, and caching LLM responses at the application layer to avoid calling the API at all. When you stack all three levers together, compound savings of 80–95% are realistic for production workloads. The key insight is that these levers are independent — you do not need to choose one over the other. Start with the lowest-effort lever for your architecture, measure the impact, then layer on the next.

Prompt Cache: the biggest single savings

Prompt caching delivers a 90% discount on cached input tokens across all major providers, making it the single highest-impact optimization for most workloads. The mechanism is straightforward: when multiple requests share a common prefix — such as a system prompt, few-shot examples, or a RAG context template — the provider caches that prefix and charges a fraction of the full input price on subsequent requests. The break-even point is low: if more than 20% of your input tokens are cacheable and reused across requests, prompt caching pays off immediately. Consider a concrete example: a 4,000-token system prompt sent with every request across 100,000 monthly requests generates 400 million input tokens just for the system prompt alone. Without caching on GPT-5, those 400M tokens cost $500 per month at $1.25 per million tokens. With prompt caching enabled, the same tokens cost approximately $55 per month at the cached rate of $0.125 per million tokens. That is $445 per month in savings — $5,340 per year — from a single configuration change that takes minutes to implement. The savings scale linearly: double your request volume and double your savings. Most production applications have cacheable prefixes well above the 20% threshold because system prompts, safety instructions, and output format specifications are repeated verbatim on every call.

Batch API: 50% off for async workloads

The Batch API offers a flat 50% discount on all token costs in exchange for a 24-hour response window. Instead of receiving responses in real time, you submit a batch of requests and retrieve results within 24 hours. This tradeoff is perfect for workloads that do not require immediate responses. Ideal use cases include nightly report generation where results are consumed the next morning, document processing pipelines that ingest and summarize large volumes of files on a schedule, evaluation and testing runs where you benchmark model quality across hundreds or thousands of test cases, bulk classification tasks such as categorizing support tickets or tagging content, and data extraction workflows that pull structured information from unstructured text. The Batch API is explicitly not suitable for real-time chat applications where users expect instant replies, user-facing responses in interactive products, or time-sensitive alerts and notifications that must fire within seconds. Implementation is straightforward: instead of calling the completions endpoint per request, you package requests into a JSONL file, submit the batch, and poll for completion. Most teams find that 20–40% of their total LLM request volume qualifies for batch processing once they audit their workloads, yielding an immediate 10–20% reduction in total API spend with minimal engineering effort.

Model routing: use expensive models only when needed

A model router classifies each incoming request by complexity and routes it to the cheapest model capable of handling it well. The implementation pattern has two steps: first, classify the request complexity using either a cheap model like GPT-5 Nano or a heuristic based on token count, keyword presence, or task type; second, route the request accordingly. Simple requests — FAQ lookups, basic classification, short extractive answers — go to Nano at $0.05/$0.40 per million tokens. Medium-complexity requests — summarization, moderate reasoning, structured data extraction — route to Mini at $0.25/$2.00 per million tokens. Only genuinely hard requests — complex multi-step reasoning, creative generation requiring nuance, tasks where quality is critical — hit GPT-5 at $1.25/$10.00 per million tokens. In a typical production distribution, 60% of requests are simple, 30% are medium complexity, and only 10% truly require the flagship model. The math is compelling: if you route everything through GPT-5, your effective output cost is $10.00 per million tokens. With intelligent routing at that 60/30/10 distribution, your blended effective output cost drops to approximately $2.50 per million tokens — a 75% reduction. The router itself adds negligible cost since the classification call uses minimal tokens on the cheapest model. Teams that implement routing typically see 60–80% cost reductions while maintaining quality on the requests that actually need it.

Response caching and deduplication

Application-layer response caching eliminates redundant LLM calls entirely, saving 100% of the cost for each cached response served. Analysis of production workloads consistently shows that 15–25% of user queries are either exact duplicates or near-duplicates of previous queries. By implementing a response cache using Redis, Memcached, or even a simple in-memory store, you can intercept these repeated queries before they reach the LLM API. For exact-match caching, hash the prompt and check your cache store — if a match exists, return the cached response instantly with zero API cost and near-zero latency. For semantic deduplication, compute an embedding of the incoming query and perform a similarity search against your cache. If the new query exceeds a 95% cosine similarity threshold against a cached query, return the cached response. This approach saves approximately 20% of total LLM calls with zero quality impact because you are returning the exact same response the model would have generated. The additional benefit is dramatically reduced latency for cached responses — from seconds to milliseconds. Implementation cost is low: a Redis instance, an embedding model for semantic matching, and a TTL policy to expire stale responses. Set TTL based on how frequently the underlying data changes: static knowledge can be cached for days, while dynamic data might need hourly expiration.

Measuring optimization impact

You cannot optimize what you do not measure. Establish a cost observability practice by tracking these core metrics on a daily and weekly basis: cost per request broken down by model and endpoint, cost per active user to understand unit economics, cost as a percentage of revenue to ensure your AI features remain profitable, cache hit rate for both prompt cache and application-layer response cache, batch utilization rate showing what percentage of eligible requests actually flow through the Batch API, and model routing distribution confirming that your router is correctly classifying request complexity. Set up automated alerts that fire when cost per user exceeds your budget threshold — for example, if your target is $0.50 per active user per month and the trailing average crosses $0.60, investigate immediately. Build a weekly cost dashboard that breaks down spending by model, by product feature, and by user segment. This visibility reveals which features are cost-efficient and which are burning budget. Common findings include: a single verbose prompt in one feature consuming 40% of total spend, a feature with low usage but high per-request cost that should be moved to batch processing, or a cache hit rate that dropped because a deployment changed the system prompt format. Review the dashboard weekly with your engineering team. Optimization is not a one-time effort — it is an ongoing practice that compounds as your product scales. Teams that measure consistently find new savings opportunities every quarter as usage patterns evolve and new provider pricing tiers become available.