AI/ML6 min read

AI Cost Optimization

Reduce LLM API costs with prompt caching, model tiering, token counting, batching, and caching at the application layer.

llmcostoptimizationai

LLM API costs scale directly with usage, and costs at scale can reach thousands per month without optimization. Systematic cost optimization-model selection, caching, batching, and prompt efficiency-can reduce costs 50-90% without quality loss.

Measure before optimizing. Know your cost breakdown by feature. Use the smallest model that meets quality requirements. Cache aggressively at every layer. Batch non-real-time requests. Prompt caching is often the highest-ROI optimization.

Workflow

Measure and attribute costs by featureYou can't optimize what you don't measure; attribution reveals which features drive the most cost.
Apply model tiering based on task complexityUsing GPT-4o for summarization when GPT-4o-mini suffices wastes 10× the budget.
Enable prompt caching for repeated system promptsLong system prompts (RAG context, instructions) repeated on every call waste tokens; caching eliminates ~90% of that cost.
Implement semantic caching for repeated queriesUsers often ask similar (not identical) questions; semantic caching serves cached responses for semantically similar queries.
Batch non-urgent requestsOpenAI Batch API and similar offerings process requests asynchronously at 50% cost reduction.

Output Format

Cost dashboard by feature, model routing logic, prompt caching enabled, semantic cache layer, batch processing for non-real-time work, monthly cost target met.

Examples

Model router: `const model = task === "simple_extract" ? "gpt-4o-mini" : task === "complex_reason" ? "gpt-4o" : "claude-haiku-4-5-20251001"; // 10x cost difference`

Anthropic prompt cache: `{ "role": "system", "content": [{ "type": "text", "text": systemPrompt, "cache_control": { "type": "ephemeral" } }] }` // 90% discount on cache hits

Token counting: `import { encoding_for_model } from "tiktoken"; const enc = encoding_for_model("gpt-4o"); const tokens = enc.encode(text).length; const cost = tokens * 2.50 / 1_000_000`

Edge Cases & Gotchas

Semantic cache staleness: cached responses can become outdated when underlying data changes; use shorter TTLs for time-sensitive content and cache invalidation on data updates
Prompt caching warm-up: cache must be primed on first request; cold starts don't benefit; ensure high-traffic use cases keep cache warm
Quality vs cost trade-off: document explicitly where you chose a smaller model for cost; set up evals to catch if quality degrades over time

When to Break These Rules

For safety-critical applications, never trade model quality for cost. Use full-capability models for medical, legal, or high-stakes decisions regardless of cost.

Reference Files

Anthropic prompt caching: https://docs.anthropic.com/claude/docs/prompt-caching
OpenAI Batch API: https://platform.openai.com/docs/guides/batch
GPTCache: https://github.com/zilliztech/GPTCache
LLM pricing comparison: https://artificialanalysis.ai/