A junior added a cache to speed up the slow endpoint.
The endpoint was fast. The data was wrong.
The senior developer asked, “Did the cache work?”
“It did exactly what I asked it to.”
“And was that what you wanted?”
The Code Behind the Koan
Almost every caching incident starts with a correctly implemented cache. The code is fine. The TTL is set. The keys are sensible. The endpoint drops from 900ms to 12ms and the dashboard turns green. Nothing failed. That is what makes this failure so hard to see coming.
The junior’s answer is honest and complete. The cache did exactly what it was asked to do, which was to hold a value and hand it back without checking whether the world had moved on. The senior’s follow up separates two things engineers collapse together every day. There is the request you wrote down, and there is the outcome you actually wanted. A machine will give you the first one with total obedience and no interest whatsoever in the second.
Cache invalidation earns its reputation as one of the hard problems in computing because it is not a technical question at all. It is a question about tolerance. How stale is acceptable, for which piece of data, and who pays when the answer turns out to be wrong? A price that is 30 seconds old is usually harmless. An inventory count that is 30 seconds old can sell the same unit twice. The code looks identical in both cases.
Notice too that the slow endpoint never got fixed. It got hidden. A cache in front of an expensive query is a decision to stop asking why the query is expensive. Sometimes that is the right call, made deliberately, with the reason written down. Often it is a way of buying quiet, and the debt comes due later when the underlying query becomes slow enough that even the cache misses hurt.
Before your next cache goes in, finish this sentence in the pull request description: this data may be up to N seconds out of date, and here is who that hurts. If you cannot finish it, you do not yet know what you wanted. You only know what you asked for.