لا تثق في KV Cache: الطرد العملي، الأصول، والقياس لحوسبة الوكلاء

The KV Cache Is Not Harmless State

Agent stacks treat the key–value cache like a friendly notepad. You prefill a model, carry forward its attention context, and hope the shortened path to an answer saves both time and money. That belief feels efficient. It also smuggles in risk.

The KV cache is not memory in any human sense. It is a substrate of attention: matrices distilled from tokens and prompts, a set of intermediate activations that shape what the model will pay attention to next. Treating it as harmless state is like treating a database’s write-ahead log as a notebook. It is a sharp object. Touch it without a plan and you will cut yourself.

Two operational myths drive today’s misuse. The first: that caches are automatically safe because they live inside vendor runtimes and disappear after requests. The second: that reuse is always a win, because saved compute is the only metric that matters. Both assumptions break in production. The cache influences what the model sees as relevant. It can encode sensitive details. It can amplify errors. It can be reused where it should not be, across tools, tenants, or altered policies. It can silently dominate the model’s next step in a way you cannot see in the tokens.

This piece takes a side: do not trust the KV cache. Make it conditional. Make it provenance-aware. Meter it for cost and for privacy. Teams that do this will prevent reasoning drift, avoid costly leaks, and keep budgets defensible. Teams that do not will watch their agents grow unpredictable and their bills slope upward.

Where Corruption and Leakage Begin

“Corruption” sounds dramatic. In practice it looks mundane: small shifts in weight allocation that compound into different tool selections, different citations, or different safety outcomes. You take a plan from a previous turn, save the cache, and reuse it for the next call. The model pays a little extra attention to a stale instruction or a now-irrelevant snippet fetched from a retriever. The error surface changes. A choice that would have been borderline is now favored. Multiply that by dozens of hops and you have a different agent.

The most common failure modes show up in three places.

First, tool selection and scratchpad carryover. Many stacks keep a cache across the system prompt, current user question, and the agent’s last few tool calls. That can help avoid restating the rules. It can also whisper an old plan into the model’s ear. The model leans toward a tool it used earlier, simply because the cache asks it to. An agent meant to re-evaluate each turn now has a prior baked into its attention. When the environment changes—APIs degrade, schemas evolve, or the user’s goal shifts—the agent reacts too slowly or not at all.

Second, retrieval contamination. Retrieval-augmented generation produces tokens from a blend of user input and external documents. If you encode those into the cache and let them ride, you carry forward latent patterns from a specific document into contexts where it does not belong. With adversarial or just poorly curated corpora, you risk prompt injection by accumulation: harmless on one call, decisive on the next. The cache has no natural sense of scope. Without a boundary, it remembers the wrong thing too long.

Third, cross-entity leakage. Even if your code keeps per-user contexts separate, the operational line between tenants, projects, or regulatory jurisdictions can be thin. Caches being shared across microservices, or a “global” KV reuse layer intended for efficiency, can bridge those lines. What leaks is not a string of text; it is an influence vector. The next generation sees a bias that came from someone else’s instruction or data. In regulated environments, intent and mechanism both matter. A regulator will not be comforted by “it was only an activation.”

There is also the simple cost trap. Caches change the marginal cost curve. A cheap cache makes extra steps feel free, so orchestration layers add hops. Speculative decoding, branchy tool calls, and longer chains become fashionable. Then the base models get larger, the cache windows get wider, and the spend creeps. The savings that helped win the internal argument last quarter become the reason the budget is red this quarter. Hidden state invites hidden cost.

These are not hypothetical. You can reproduce them on your own stack with some disciplined experiments. Freeze your prompts and tools. Run identical tasks twice: once with reuse across turns, once without. The divergence will not always be dramatic, but you will see pockets where the cached run drifts or prefers stale context. That is the footprint of a policy gap.

Conditional Reuse With Provenance, Not Faith

The only sane default is to treat the KV cache as untrusted until it proves itself relevant again. That means tying reuse to the actual conditions that make reuse valid, and encoding where the cache came from. It also means being clear about what you will not reuse.

Start by defining provenance at the segment level. Do not think of the cache as a monolith. Partition it into logical segments aligned to the natural seams in your prompt: the immutable system contract, the current user turn, each retrieved document block, each tool’s input and output wrapper, the scratchpad. Attach metadata to each segment. At minimum record source type, sensitivity label, time-to-live, and scope. Sensitivity tags should reflect who owns the content and whether it is derived from personal data, customer IP, or public material. Scope should encode the permissible reuse boundary: same turn, same session, same user, same project, or cross-project.

Define a reuse key for the entire cache and for each segment. The key must hash the full set of conditions that materially affect interpretation: model and tokenizer version, system contract fingerprint, safety policies, active tools and their schemas, the sampling regime, and any attention modifications in use. If any of those change, segments dependent on them become invalid by default. This one rule stops a large class of accidental misreuse when engineers silently upgrade a prompt or rotate a tool.

Then make reuse conditional on both stable inputs and current relevance. Stable inputs are the ones captured in the reuse key. Relevance you measure at request time. You do not need perfect math to improve outcomes; you need conservative rules grounded in signals the model already gives you.

One practical signal is surprisal change. Given the new user turn, score the previous-turn segments by how “surprising” they look to the model’s token distribution for the next step. If a retrieval segment sits far off the expected path, gate it. Treat that segment as cold. Let the model recompute attention without it. This keeps the cache from dragging unrelated documents into the next step simply because they are already encoded.

Another signal is reuse agreement. Generate a cheap, low-precision plan without the cache—just the system prompt and the current turn—and compare its tool choice or outline against a cached version. If they disagree beyond a threshold, quarantine segments that tilt the choice. This can be as simple as retaining system and instruction segments while excluding older scratchpad content that was planning-heavy. You are not trying to out-math the model. You are sanity checking whether yesterday’s plan should drive today’s act.

You can also use randomized attention to reduce dependence on specific cache positions. Recent experiments suggest that adding mild, structured randomness to attention patterns can improve generalization and reduce overfitting to specific context tokens. Apply that thinking to cache reuse: when you do carry segments forward, sample a small fraction of heads or positions to be dropped from reused segments. You will see a tiny penalty in variance and sometimes in speed. You will also limit the cache’s ability to hard-wire a path. It is the same principle as dropout in training, applied to reuse in inference.

Combine these into a policy you can explain:

– Always reuse the system contract and current turn segments if and only if the reuse key matches.

– Reuse tool wrappers and schemas only if the tool set and versions match. Disallow reuse of raw tool outputs by default. Summarize them into semantically bound notes instead.

– Score retrieval segments for surprisal against the current turn. Keep only the top few that remain on-distribution; age the rest out.

– Treat scratchpad segments as ephemeral. Do not reuse them across turns without a summarization step that produces a compact, human-auditable state of facts and open questions.

– Respect scope. If a segment is scoped to a session or a user, the policy engine should reject reuse requests from a different scope, even if a misconfigured service accidentally requests it.

Notice what this avoids. You are not baking arbitrary TTLs into everything. You are tying reuse to the conditions that make it coherent, and you are grounding those conditions in both system fingerprints and the model’s own local signals. That balance keeps the policy legible to an auditor and explainable to your team.

Metering: Cost, Privacy, and the Case for Budgets

Caches are easy to add and hard to pay for. The only way to tame cost is to meter it at the same granularity that you meter inference. That begins with a clean definition: track not just tokens in and out, but cache hits and their effective token-equivalent, per segment and per scope. A hit that saves 300 tokens of compute but drags an irrelevant retrieval segment forward did not save you money; it shifted the spend into error correction and tool retries.

Build a cost ledger. For every request, log the segments reused, their estimated token-equivalent, the latency saved, and the downstream tool calls triggered. Roll that up by agent, by project, and by user population. Then attach a budget. Make it explicit that cache-enabled steps have a monthly ceiling, just like base inference. If you work in a firm where budget discipline matters, this is the only way to keep savings real and not imaginary.

Now do the same for privacy and data exposure. Treat reuse as an exposure event. Every time a segment is reused, increment an exposure count under the owning identity and scope. Tie it to time. If a segment classified as personal data is reused a certain number of times, require re-validation or re-summarization. If it crosses a time threshold, require re-consent or purge. This is not differential privacy; you are not adding noise or proving bounds. You are building a practical control that stops the quiet creep of sensitive content into every corner of your agent’s attention.

In regulated environments, this exposure ledger needs to align with data residency and multi-entity rules. A Gulf conglomerate with entities in different jurisdictions cannot have a cache that jumps borders just to save milliseconds. Encode residency into scope. Encode entity boundaries into access control. If your cache provider cannot support that, do not use cross-request reuse for regulated data. Keep it local and ephemeral.

Finally, introduce kill switches. When costs spike, or when an exposure metric trips, you should be able to turn off reuse by segment type, by project, or globally within minutes. Not days. Not a sprint later. If your architecture requires a redeploy to change a reuse policy, you are not in control of your spend or your risk.

An Architecture That Treats Caches Like Data

A reliable posture treats the KV cache as a governed data product, not an incidental detail of inference. The architecture that follows looks familiar if you have built systems around event logs and content-addressed storage.

Start with a cache service that is separate from your orchestrator and from the model runtime. The orchestrator asks for reuse; the cache service decides. That separation lets you enforce policy even when product engineers feel pressure to “just speed it up.” It also gives you a place to centralize metering and audit logs.

Represent caches as segment bundles. Each bundle contains typed segments with metadata, provenance, and a content hash. The content hash is not the raw token text; it is a deterministic fingerprint of the embedded representation under the specific model and tokenizer. If the model changes, the fingerprint changes, and reuse must be re-evaluated. Store these bundles with encryption at rest and track an access-control list that reflects the scope rules. When an orchestrator asks to reuse a bundle, the service checks the reuse key, scope, and metering budget before handing back a handle.

Do not ship the raw activations between services unless you must. Where vendors allow pinned caches within a session, prefer local reuse and keep the provenance logic in-process. Where you need cross-request reuse, materialize only the minimal information needed to reconstruct the relevant segments. If the platform gives you a structured interface for cache handles and resumption, keep that abstraction boundary clean. Resist the temptation to roll your own binary blob handoff.

Build a summarization layer for scratchpads and tool outputs. A “commit” step transforms ephemeral thought into compact state. This summary is what you persist across turns, not the raw activation pattern that captures the entire chain-of-thought. The summary should be reviewable and testable. It is cleaner to carry forward a short set of facts and constraints than a kilometer of internal tokens with unknown biases.

Integrate the policy engine alongside the cache service. The engine needs three inputs: a fingerprint of the current run, a set of signals about relevance and surprisal, and the operator’s policy configuration. It decides which segments are eligible and with what sampling of attention to apply. It also emits an explanation string. Not a novel. A short record of what it kept and why. When someone asks why an agent made an odd choice, you want to be able to say: the reuse key changed, so prior tool outputs were excluded; or, three retrieval segments were gated for low relevance and only two were kept.

Place your metering pipeline adjacent. The ledger collects per-request entries for hits, misses, token-equivalent savings, exposure increments, and any budget flags. Feed that into your normal observability stack. The caches do not deserve a bespoke dashboard culture. They deserve the same SLOs and escalation paths as your other stateful systems.

Mind the network. Do not swing bundles across regions just to chase shared reuse. Latency and residency both degrade when a cache service becomes centrally located far from inference. Keep caches physically close to the model endpoints you call. Where that is not possible, segment strictly and prefer recomputation over remote reuse. In markets like the UAE where data jurisdiction and private-cloud footprints are top of mind in government and finance, this matters not only for law but also for common sense.

None of this requires exotic infrastructure. It does require a new habit of treating inference state with the same respect as application data. Teams that build this way stop shipping raw thought. They ship facts. They keep the rest where it belongs: ephemeral and local.

Testing and the Counter‑Case

You can adopt the cleanest policy and still ship a brittle system if you do not test the right failure modes. Testing here is not about making the outputs pretty. It is about measuring drift, contamination, and the true value of reuse.

Begin with invariants. If you disable cache reuse while holding prompts, tools, and seeds constant, your task outputs should remain within an acceptable band. Define that band early and make it observable. For retrieval tasks, the band might be overlap in cited sources. For tool-using agents, it might be agreement on tool sequence and parameters. If running without reuse produces wildly different outcomes, your agent depends too much on cached priors. That is a policy smell.

Run cold–warm A/Bs as a matter of routine. Every deployment should include a slice of traffic that runs with and without reuse. Compare latency, cost, and quality metrics. Look past the headline “saved tokens.” Track downstream effects: retries, escalations to larger models, tool failure rates. A cache that saves tokens but causes more tool churn is not a savings machine. It is a cost shifter.

Inject chaos into the cache path. Randomly drop segments marked as low-importance to ensure graceful degradation. Rotate the reuse key to simulate a model upgrade and watch how the system behaves. An agent that craters during a model change was never robust; it was memorizing.

Probe for contamination with adversarial content. Seed your retriever with documents that include misleading instructions and watch whether they linger in reuse. If your gating suppresses them on the first pass but they reappear later through cached attention, your policy is missing a fence.

Audit provenance. Sample a set of production requests and manually trace which segments were reused and under what scope. If you cannot reconstruct that path, you do not have an auditable system. If the path shows segments crossing entity lines or residency boundaries, you have a bigger problem than accuracy.

Now the counter-case, stated with as much charity as it deserves. Some argue that the KV cache is ephemeral by design and therefore safe. It lives in vendor memory, not your database. It is vendor-encrypted. It is bound to a session. Any extra rules only add latency and complexity. And for many tasks, reuse works well enough: it speeds up multi-turn chat, it reduces redundant tokenization of static instructions, and it makes small models act a little bigger. Why overthink it?

There is a grain of truth in each claim. Local, within-request reuse is often a win and belongs in your default path. Vendor-managed session caches can be safe when you keep the scope tight and your prompts stable. For short-lived assistants with no sensitive inputs, the risk profile is mild. If that is all you build, do not build a cathedral of policy around something that does not need it.

But the argument breaks in two places. First, “ephemeral” says nothing about appropriateness. A bad influence that dies after a minute is still a bad influence. If it tilts a decision in that minute, the damage is done. Second, “works well enough” is only valid when the boundary conditions do not change. The moment you add tools, RAG, multi-agent orchestration, or multi-tenant traffic, the implicit assumptions snap. A cache that felt like a speed hack becomes a shared state you cannot explain or audit. Hidden state is where systems rot.

There is also a growing body of research and practice pointing to conditional reuse and randomized attention as stabilizers. You do not need to quote specific papers to grasp the thrust: models get brittle when they over-commit to a narrow context, and light randomness plus condition checks help them adapt. Product experience matches that. Teams that experiment with conditional gates tend to report fewer unforced errors and a flatter cost curve. Those who do not keep adding patches.

So keep the kernel of the counter-case: do not stretch policy farther than your risk requires. Then drop the rest. Treat caches like the sharp objects they are. Keep them wrapped. Pick them up with intent.

Stop Trusting It; Start Governing It

Agentic reasoning is seductive. You add a planner, a scratchpad, a retreiver, and a cache, and suddenly the system behaves with momentum. It remembers what it just thought. It acts with confidence. That is also how drift settles in: the system starts to believe its own drafts.

A cache is not a mind. It is a memory of weights, the afterimage of a previous glance. If you would not persist a piece of data in your database without a schema, a lineage, and a policy, do not persist its influence in your model without the same. Provenance-aware eviction is not ceremonial. It is how you keep agents from spiraling into stale plans. Conditional reuse is not fancy. It is table stakes for correctness. Metering for cost and privacy is not bureaucracy. It is hygiene.

Treat the KV cache like a write-ahead log for attention. Summarize what matters. Expire what does not. Gate the rest. And tie every reuse to a reason you can say out loud. If your team cannot explain why a particular segment stayed alive, it should not have. The cost of building this posture is small compared to the cost of letting cached thought steer the next thought without anyone watching.

Stop trusting the cache. Govern it. That is how you keep agents sharp, safe, and within budget, long after the first demo glow has faded.