One Line of Code That Can 10× Your Agent Costs: KV-Cache-Friendly Agent Design

Ever run into this situation: your Agent works functionally, yet it runs slow and costs a fortune. First‑token latency fluctuates wildly, and end‑of‑month bills are shocking.

More than likely, your Agent is blowing up the KV Cache.

I once saw a customer‑service Agent case: a line Current time: 2026‑xx‑xx xx:xx:xx was appended at the end of the system prompt. Just this single line made every request prefix different, keeping cache hit rate stuck in single digits. After moving the timestamp into a user message at the end of the conversation, hit rate soared to 85 %, first‑token latency dropped by 40 %, and costs were slashed drastically.

This is all KV Cache at work.

This article breaks down KV Cache thoroughly: what it is, why prefixes must stay unchanged, how to structure context, and common pitfalls to avoid.

1. What is KV Cache, and why are prefixes so sensitive

KV Cache is the model's "scratch pad"

For every new token a large model generates, it matches its query against the keys of all preceding tokens, then computes a weighted sum over the values. Without caching, every new token requires recomputing all prior tokens. Computation cost grows rapidly as context length increases.

KV Cache does one simple thing: store previously computed keys and values for later reuse. Only the new token's KV is calculated; everything before is "copied from scratch‑pad".

Changing one character in the prefix invalidates all subsequent cache

Attention has a hard constraint: each token's KV depends only on tokens that come before it. In other words, alter even one character in the prefix, and all cache from that point onward becomes invalid and must be recomputed.

Cross‑request caching: the cost‑saving scratch pad

KV Cache operates within a single request. By contrast, OpenAI Prompt Caching, Claude Prompt Caching, and DeepSeek Context Caching persist this cache across requests. Read cost is roughly 1/10 of the original computation cost.

Therefore, stable prefixes reduce both latency and cost.

Below are three golden rules for achieving stable prefixes.

2. Three Golden Rules: fundamentals for cache‑friendly design

2.1 Golden Rule #1: Never modify fixed system prompts and tool definitions

System prompts and tool definitions sit at the very front of context and form the static prefix. Even a single space, line break, or dynamic variable change invalidates the entire cache.

2.1.1 Common bad pattern

# ❌ Cost‑inefficient pattern
system_prompt = f"""You are an intelligent customer‑service Agent.
Current time: {datetime.now()}
User ID: {user_id}
Rules:
1. xxx
2. xxx
{tool_definitions}
"""

Timestamps, user IDs, session IDs baked into the system prompt make every request prefix unique, so cache never hits.

2.1.2 Correct approach

# ✅ Cache‑friendly pattern
system_prompt = """You are an intelligent customer‑service Agent.
Rules:
1. xxx
2. xxx
""" # Fully fixed, byte‑level unchanged

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": f"User ID: {user_id}"}, # appended
    {"role": "user", "content": f"Current time: {datetime.now()}"}, # appended
]

Treat the system prompt like a constitution: once set, never alter it. Inject dynamic information by appending new messages at the end.

2.1.3 Same principle for tool definitions

Tool schemas consume large token counts. Reordering them changes their hash. Tool definitions must keep both content and order fixed. Avoid tricks such as "dynamic sorting by usage frequency"; the benefit is negligible while cache is completely destroyed.

2.2 Golden Rule #2: Always append dynamic information to the end

Appending new content to the end of the message list does not alter KV values for already‑cached tokens. Place all dynamic data at the tail:

• Current time → append as a message

• Tool call results → append as tool‑role messages

• Agent runtime states → append to the end

• User‑specific data (username, preferences, order IDs) → append to dynamic section

Note: sliding window is an anti‑pattern. It discards earliest messages while keeping the most recent N entries. This shifts positions of all remaining messages and invalidates cache. Worse, the Agent suffers memory loss: once early tool outputs slide out of window, the Agent effectively forgets them.

2.3 Golden Rule #3: Use standard API formats; do not manually concatenate messages

Some Agents convert structured role‑content messages into raw text streams:

USER: What is the weather today?
ASSISTANT: Let me check.
TOOL: Beijing, sunny, 32℃

This creates two major problems.

2.3.1 Breaks model structured parsing

Models are trained on role‑based formats. When converted to plain text, the model expends extra attention inferring speaker identity, triggering abnormal behaviours: repeated operations, ignoring tool outputs, generating text instead of invoking tools.

2.3.2 Breaks chain‑of‑thought preservation: Qwen3 example

Qwen3 natively uses ChatML. Beyond standard roles (system/user/assistant/tool), it uses … for internal reasoning and <|tool_call|> for tool invocation.

With thinking mode enabled (enable_thinking=True), model outputs are wrapped in … blocks regardless of /think or /no_think toggles. If thinking is disabled, this block is empty.

Example multi‑turn tool call. User asks "What is the weather in Beijing today?" Qwen3 (reference doc) ChatML output:

<|im_start|>system
You may call tool: get_weather(city) to fetch weather
<|im_end|>
<|im_start|>user
What is the weather in Beijing today?
<|im_end|>
<|im_start|>assistant
<|tool_call|>[{"name": "get_weather", "arguments": {"city": "Beijing"}}]<|tool_end|>
<|im_end|>
<|im_start|>tool
Beijing weather: sunny, 25℃
<|im_end|>

If code incorrectly rewrites tool result into <|im_start|>user Beijing weather: sunny, 25℃<|im_end|>, two issues arise:

1. Role misalignment: Models expect tool role for tool outputs. Forcing user role forces the model to guess message origin, leading to ignored tool results and redundant tool calls.

2. Prefix drift: … blocks are part of assistant messages. Stripping … blocks from historical assistant responses for "cleanliness" breaks prefix consistency between turns and disables KV Cache.

Correct practice: Use frameworks with native function‑call support such as Qwen‑Agent or vLLM to handle ChatML rendering. Keep tool results in tool role. Do not arbitrarily strip … blocks from history.

2.3.3 Takeaway

Use official SDKs or standard API formats. Let inference frameworks handle Chat Template conversion; avoid manual string concatenation.

2.4 Differences in handling historical chain‑of‑thought across models

Above example uses Qwen3 ChatML thinking blocks. Different models adopt distinct strategies for historical reasoning traces and require Agent‑specific handling.

2.4.1 DeepSeek: tool‑call turns must return complete reasoning_content

In thinking mode, DeepSeek returns reasoning via reasoning_content, a peer‑level field alongside content. Official documentation (link) states clear rules:

• Regular multi‑turn chat (no tool calls): reasoning_content from prior assistant messages may be omitted; the API ignores it even if supplied.

• Tool‑call‑involved turns: Intermediate assistant reasoning_content must be fully passed back across all subsequent user interactions. Missing this field triggers direct 400 errors.

Example workflow. User asks "What will the weather be in Beijing tomorrow?" DeepSeek first calls get_date, then get_weather:

Turn 1a: User: "What will the weather be in Beijing tomorrow?"
         → Model: reasoning_content="I need to fetch tomorrow's date..."
                   tool_call: get_date()
Turn 1b: Tool result: "2026‑06‑13"
         → Model: reasoning_content="Tomorrow is June 13, call get_weather"
                   tool_call: get_weather(location="Tokyo", date="2026‑06‑13")
Turn 1c: Tool result: "Rainy, 15°C"
         → Model: reasoning_content="Rainy and cool; user should bring an umbrella"
                   content: "Yes, bring an umbrella! Rain expected in Beijing tomorrow, around 15°C."

Within this tool‑call loop, reasoning_content from Turn 1a and 1b must be preserved in Turn 1c and all future user queries. The simplest implementation: messages.append(response.choices[0].message); keep the full message object including reasoning_content, content, tool_calls; do not manually prune fields.

2.4.2 Claude: thinking blocks with signatures must be returned verbatim

Claude extended thinking produces thinking‑type blocks, each carrying a signature field (official docs).

Official specified behaviour for thinking + tool‑call multi‑turn workflows:

• Model generates thinking content before issuing tool requests;

• After receiving tool outputs, the model does not repeat prior reasoning immediately;

• Next thinking block appears only upon the next non‑tool_result user turn.

Weather query example for Claude:

# First request
resp = client.messages.create(
    model="claude‑sonnet‑4‑6",
    thinking={"type": "enabled", "budget_tokens": 2048},
    tools=[weather_tool],
    messages=[{"role": "user", "content": "What is the weather in Beijing today?"}]
)
# resp.content = [thinking_block, tool_use_block]

# Second request: thinking_block AND tool_use_block must be passed back unchanged
messages = [
    {"role": "user", "content": "What is the weather in Beijing today?"},
    {"role": "assistant", "content": resp.content}, # ← Full payload, do not drop thinking_block
    {"role": "user", "content": [{"type": "tool_result", ...}]}
]

Filtering out thinking_block while retaining only tool_use in second‑request payload causes:

1. Cache impact: thinking blocks belong to context. Removing them alters prefix and invalidates KV Cache.

2. Protocol impact: Two scenarios:

   • Tool‑call workflows (strict requirement): When extended thinking is combined with tool calls, complete thinking blocks including signatures must be returned unmodified. Missing or tampered signatures get rejected by providers ("Blocks with missing or modified signature fields are rejected by the provider").

   • Plain multi‑turn chat (recommended): Thinking blocks may technically be omitted without API errors, yet official guidance recommends always returning them as the most robust approach.

Claude best practice: Persist the full content array of assistant messages. Do not flatten into simple {role, text} objects. Render display text extracted from text‑type blocks for end‑user presentation.

2.4.3 Summary

Common takeaway: Manual string building or flattening history may work for isolated single‑model single‑turn cases. Multi‑turn tool invocation and cross‑model adaptation expose flaws. Golden Rule #3 advocating standard API formats represents hard‑earned engineering consensus from error logs, request rejections and cache failures.

3. Context structuring: stable prefix + dynamic suffix

3.1 Four‑section template

Applying the three golden rules, Agent prompt layout:

[System]
System prompt (fixed)

[Tools]
Tool definitions / schemas (fixed)

[Examples]
Static few‑shot examples (fixed)

[History + Current]
Dynamic conversation history + current query (appended at tail)

First three segments form stable prefix; last section holds dynamic delta. Cache boundary sits between section 3 and 4.

Full computation occurs on first turn. Subsequent turns compute only incremental new tokens while hitting cache for all prior content.

3.2 Handling long conversations: summarization + truncation, avoid sliding window

Context length will eventually overflow. Correct mitigation: summary compression plus rolling truncation.

[summary] User asked about refund procedure; assistant provided return address. ← Compressed summary
[recent] User: I still have not received my refund. ← Full recent turns preserved
          Assistant: Please provide your order ID.
          User: Order ID is 12345.

Generate summaries deterministically and cache summary outputs to avoid recomputing every round. When rolling‑truncating, drop an entire chunk from the very front while preserving ordering of remaining messages.

3.3 Can cache be shared across users or Agents?

Not by default, but architecturally possible. KV Cache reuse requires byte‑exact prefix matching.

3.3.1 Cross‑user sharing

All users share identical system prompt and tool definitions. No user‑specific fields injected into prefix. Providers can reuse this cache segment for all users. User‑specific attributes go exclusively to dynamic sections.

3.3.2 Cross‑Agent sharing

Within multi‑Agent systems, child Agents share a common platform‑level system prompt (security rules, general capability statements), followed by individual role definitions. This platform prefix gets reused across child Agents.

3.3.3 Counter‑intuitive conclusion

Customising system prompts per‑user seems to improve experience, yet higher customisation lowers cache hit ratio. Place customisations inside dynamic sections to retain personalisation without breaking prefix sharing.

4. Overlooked cache‑killers

4.1 Misuse of reasoning effort parameters

When reasoning_effort is passed as an API top‑level parameter, prefix stays unchanged and cache works. Baking it into system prompt e.g. You are an Agent with high reasoning effort mutates prefix and destroys cache.

Rule: Keep effort as API parameter; never embed it inside prompts.

4.2 API proxy gateways

Using proxy gateways typically drives cache hit rate near zero. Proxies inject custom system prompts, user identifiers and rewrite message schemas. The prompt seen by upstream LLM diverges from original payload hash.

Proxy‑advertised "caching" usually refers to response cache (return stored result for identical requests), not KV Cache. Agent requests are almost never identical, yielding poor hit rates.

Recommendation: Prioritize direct official API access if cache efficiency matters. If proxies are mandatory, select pure‑forwarding gateways. Periodically compare cached_tokens metrics between direct and proxied traffic for validation.

4.3 Mid‑session model switching

Switching models e.g. GPT → Claude invalidates cache even for identical inputs, as each model maintains independent KV Cache stores. Multi‑model routing Agents should pin model selection per‑task; avoid switching models within one session.

5. Measure it: build cache observability

Optimisation requires metrics. Log three metrics in production:

• prompt_tokens: tokens requiring fresh computation for this request

• cached_tokens: tokens served from cache

• cache_hit_rate = cached_tokens / (cached_tokens + prompt_tokens)

OpenAI, Claude and DeepSeek APIs expose these fields. Persist cache metrics alongside request‑ID.

KV Cache Troubleshooting Checklist

1. Are dynamic literals (timestamps, random IDs, user states) present inside prefix?

2. Is history being sorted, filtered or rewritten?

3. Are tool definitions regenerated or reordered at runtime?

4. Are line‑breaks, trailing spaces and encodings consistent?

5. Are summaries regenerated on every turn?

6. Are messages manually flattened into raw text strings?

7. Does traffic pass through proxy gateways?

8. Are models switched mid‑session?