AI & Development
Caching is one of the most effective tools for reducing LLM API costs and latency. Here is a practical guide to the three main caching strategies and when
Caching is the most impactful optimization available for many AI applications, and it operates at multiple levels that are often not exploited together. Provider-level prompt caching reduces the cost of reprocessing static prompt prefixes. Application-level exact match caching eliminates API calls entirely for repeated queries. Semantic caching extends this to queries that are different in phrasing but equivalent in meaning. Each layer captures a different set of repeat requests, and together they can dramatically reduce both cost and latency.
Provider prompt caching operates on the input to the model API call. When a request shares a prefix with a recent request, the provider reuses the cached KV representation of that prefix rather than recomputing it. This reduces input token costs by 75-90% for the cached portion, typically a long static system prompt.
Enabling prompt caching requires structuring prompts so that static content comes first and dynamic content comes last. Once structured correctly, prompt caching is largely automatic - no application-level implementation is needed. This is the easiest caching layer to capture and should be the first optimization applied.
Exact match caching stores the complete response for each unique API request (system prompt + messages hash) and returns the stored response for identical subsequent requests. This is standard application caching applied to LLM calls: Redis or Memcached as the cache store, a hash of the request inputs as the cache key, and a TTL appropriate to how quickly the relevant information becomes stale.
For applications that frequently receive the same queries - FAQ bots, customer support tools with common questions, search features with popular queries - exact match caching can eliminate a substantial fraction of API calls entirely. The limitation is that any variation in the input (different wording, different conversation history, different system prompt) produces a cache miss. Exact match caching captures identical queries, not equivalent ones.
Semantic caching extends exact match caching by using embedding similarity to match queries that are semantically equivalent even if not textually identical. When a new query arrives, you compute its embedding, search a vector store of previously seen query embeddings, and if a sufficiently similar past query is found, return its cached response rather than making a new API call.
The similarity threshold controls the trade-off between cache hit rate and response accuracy. A very high threshold (0.98+ cosine similarity) produces a low hit rate but high accuracy - only near-identical queries match. A lower threshold (0.92-0.95) produces a higher hit rate but may return cached responses that are slightly off for the new query. The right threshold depends on how consistent the user's queries are and how sensitive the application is to slightly mismatched responses.
Semantic caching adds complexity: you need an embedding model, a vector store for cached query embeddings, and logic to select and validate cached responses. Tools like GPTCache provide this infrastructure as a library. The investment is worthwhile for applications where users frequently ask semantically similar questions - customer support, documentation assistants, FAQ handling.
Cached LLM responses can become stale when the underlying information changes - the product policy changes, the knowledge base is updated, a previous response is discovered to be incorrect. Cache invalidation strategy should match the staleness risk of your content. For responses derived from a specific document version, invalidate when that document changes. For responses to general questions where the answer may evolve over time, a time-based TTL is appropriate. For responses that should never be cached (personalized responses, responses based on real-time data), disable caching at the query level.
The most efficient caching architecture uses all three layers. Provider prompt caching reduces the cost of API calls when they must be made. Application-level exact match caching eliminates API calls for identical queries. Semantic caching eliminates API calls for semantically equivalent queries. Together, these layers can reduce total API calls by 50-80% for many production applications - a meaningful improvement in both cost and latency that compounds as usage grows.