Semantic Caching for AI Agents: learn the measurement, distrust the savings figure

Two customers type “How do I change my password?” and “I forgot my password, how do I reset it?”, and your agent calls the model twice to produce the same answer.

DeepLearning.AI’s short course Semantic Caching for AI Agents teaches you how to skip the second call. For an FDE, though, the most valuable part lies elsewhere: knowing when that skipped call hands the customer a wrong answer.

Picture a familiar situation: a client asks whether the model bill can come down and whether the agent can get faster. A cache is the most obvious answer. Without measurement, it is also the quickest way to lose the trust of end users.

Who teaches it, and why listen selectively?

This is an Intermediate-level Short Course, 1h24m long according to the course page, made with partner Redis. It is aimed at developers and ML engineers already comfortable with Python, embeddings and basic LLM applications; no prior knowledge of caching is needed.

Both instructors are from Redis: Tyler Hutcherson, who manages the Applied AI Engineering team, and Iliya Zhechev, a Senior Research Engineer. There is nothing wrong with a vendor teaching its own tool, provided you can separate the general principles from the product promotion.

Reusing answers by meaning, not by wording

The mechanism is compact. A query is turned into an embedding and compared by cosine similarity with stored queries. According to Redis’s blog, if the similarity exceeds a threshold, typically in the 0.85-0.95 range, the system returns the stored answer and does not call the LLM.

The fixed cost of this approach is often overlooked. Each vector search adds 5-20ms; in return, a hit saves 1-5 seconds. On a miss, the user still waits the full LLM time plus that overhead, so a cache only pays off when enough queries repeat.

Precision matters more than hit rate

How the course allocates its time shows where the emphasis lies. Of its 7 videos, the three longest are “Measuring Cache Effectiveness” (13 minutes), “Enhancing Cache Effectiveness” (12 minutes) and “Fast AI Agent with Semantic Cache” (16 minutes): 41 minutes spent on measuring and improving, and only then on wiring the cache into an agent.

The course teaches three metrics: hit rate, precision and latency. Hit rate tells you how much you save. Precision tells you how many of those hits returned the right answer; the rest returned the wrong one.

Imagine 10,000 customer-support queries at a threshold of 0.90 with a 30% hit rate, meaning 3,000 model calls avoided. If 4% of those hits are false positives, 120 people received an answer to a question they did not ask.

Now suppose you raise the threshold to 0.95, the hit rate drops to 20% and false positives fall to 1%. You are down to 2,000 savings, 1,000 fewer than before. In exchange, wrong answers fall from 120 to 20.

The numbers in this example are hypothetical, but the trade-off is real, and only the client’s logs will tell you where the balance lies. Redis recommends starting at a threshold of 0.90-0.95 and tracking false positives. If that rate goes above 3-5%, tuning the threshold is no longer enough; the architecture has to change.

The first job at a client is therefore concrete: take a sample of real logs, label pairs of queries as same-meaning or different-meaning, measure both thresholds as above, then choose by precision. Do not pick a threshold by feel during a demo.

When the data changes, the cache must change too

Demos rarely touch the problem of changing data. For fast-moving data such as prices or inventory, Redis suggests a TTL of 5-15 minutes, or clearing the cache as soon as the source data changes. Otherwise the agent will confidently quote a price that is no longer valid.

Semantic caching does not replace prompt caching either. Prompt caching is for large, fixed context; semantic caching is for repeated queries with the same intent. Redis calls combining the two layers “double caching”, and it is worth considering when a client has both a long system prompt and many repeated questions.

What to skip, and in what order to learn?

The thing to skip is the headline number. Redis writes that teams using semantic caching typically cut LLM costs by “50% or more”, depending on how often queries repeat. That is a figure the vendor states itself, not an independent measurement, so promise the client a measurement on their logs, not a percentage.

As for order, you can watch the videos in sequence, but rewatch the two on measuring and improving effectiveness before doing the optional 10-minute project. Then read Redis’s blog post comparing prompt caching with semantic caching to see the full picture.

For your career, a CV line such as “built a semantic cache, chose a 0.92 threshold based on precision measured on a labelled set of query pairs, kept false positives below 3%” carries far more weight than “experienced with Redis”. The numbers in that line must be ones you measured yourself.

The course teaches you how to make an agent faster. When a client asks “how much will we save?”, the answer can only come from their own logs.