AI & Development
LLM API costs can scale faster than your user base. Here are the practical techniques for controlling AI spending while maintaining application quality.
LLM API pricing is typically per token - you pay for every token you send (input) and every token the model generates (output). At small scale, these costs are negligible. At production scale, with thousands or millions of API calls per day, token costs become a meaningful business expense. Understanding where tokens go and how to reduce them without degrading quality is a core competency for teams shipping AI products.
The first step is instrumentation. Log every API call with its token counts - input tokens, output tokens, and the model used. Aggregate these by endpoint, by user type, and by time of day. Without measurement, you are guessing about where costs are coming from. With measurement, you can identify the specific call patterns that drive the most spending and prioritize optimizations by expected impact.
Most teams discover that a small fraction of calls account for a large fraction of costs. A long system prompt repeated on every call, a RAG pipeline that injects too many retrieved chunks, or a feature that generates unnecessarily long outputs - these are the patterns that cost-conscious measurement reveals.
Using the most capable model for every task is expensive and often unnecessary. Frontier models (GPT-4o, Claude Opus, Gemini Ultra) are the right choice for tasks that require their full reasoning capability. For simpler tasks - classification, short extraction, straightforward question answering - smaller, cheaper models (GPT-4o-mini, Claude Haiku, Gemini Flash) perform nearly as well at a fraction of the cost.
Routing is the strategy that formalizes this: classify incoming requests by complexity and route to the appropriate model tier. A simple factual question goes to a small model. A complex analytical task goes to a large model. Getting routing right requires evaluating which tasks actually need the large model's capability - the answer is usually "fewer than you thought."
Most major providers now offer prompt caching: if the beginning of your prompt (the system prompt, or a long document that appears in every call) is identical across calls, the provider caches the processed key-value tensors for that prefix. Subsequent calls with the same prefix do not re-process those tokens - you pay a lower "cache read" price instead of the full input price. Cache reads are typically 50-90% cheaper than uncached input tokens.
Maximizing cache hit rates requires designing your prompts to keep the cacheable prefix stable. Dynamic content (user-specific context, current date, per-call variables) should appear at the end of the system prompt rather than at the beginning, so the static prefix remains unchanged across calls and stays in cache. This simple structural change can reduce input token costs by 50-80% for applications with long system prompts.
Output tokens are typically more expensive than input tokens and are also the variable that most impacts latency. Controlling output length is one of the most effective levers for both cost and speed. Instructions like "Be concise. Limit your response to 3 sentences unless the question requires more detail" directly reduce token consumption. For structured output tasks, limiting the number of items returned or truncating less important fields reduces output length predictably.
Setting a max_tokens parameter caps output length as a hard limit. This is useful for preventing runaway generations - cases where the model generates much longer outputs than expected due to prompt design or unusual inputs. Combining a system prompt instruction to be concise with a max_tokens limit gives you both the model's best effort at brevity and a safety net.
For queries that are frequently repeated - popular questions, common lookups, queries that are identical across users - caching the LLM response at the application level eliminates API calls entirely for those queries. A semantic caching layer (embedding the query, searching for similar past queries, returning cached responses for matches above a similarity threshold) extends caching beyond exact-match responses to semantically similar ones.
Semantic caching requires evaluating the trade-off between cache hit rate (higher with a lower similarity threshold) and response accuracy (lower cached responses may not quite match the current query). For applications with many similar queries - FAQ systems, documentation assistants, product recommendation chat - semantic caching can eliminate 30-60% of API calls at the cost of occasional slightly mismatched responses.
Most LLM providers offer batch APIs for processing large volumes of requests asynchronously at reduced cost - typically 50% of the real-time price. For use cases where responses are not needed immediately - nightly content generation, batch analysis of historical data, pre-processing documents - batch API reduces costs significantly with no quality trade-off. The constraint is latency: batch jobs complete on the provider's schedule (typically within minutes to hours) rather than immediately.