KV caching is not a knob you tune. It is a prefix match — and its efficiency is decided by a single number: the position of the first token that changes. This is a forensic map of those boundaries, drawn from how Hermes Agent actually assembles its prompt.
Drag the divider. Move a volatile block one position later and the cache-hit prefix grows — the recomputed tail (and its cost) collapses. That is the entire discipline.
A decoder transformer attends every new token to all previous tokens. The Key/Value projections of those past tokens are computed once and stored — the KV cache. Provider "prompt caching" is the same idea at provider granularity: the provider stores the KV state of a token prefix and bills a cheap cache read instead of a full prefill.
The consequence is absolute. A cache hit requires an exact match from token 0 to token N. The first divergence invalidates every token after it. So efficiency is never "how much did I cache" — it is where does the first changed token sit. Everything downstream of that position is recomputed at full price.
Hermes builds the cached system prompt as three ordered tiers, joined by blank lines. The ordering is the optimisation: the most durable content is written first, so the byte sequence survives longest against the prefix rule. Nothing is re-rendered mid-session.
SOUL.md), tool & model guidance, execution guidance, coding brief, platform hint, pinned skills.system_message, project files (.hermes.md → AGENTS.md → CLAUDE.md → .cursorrules), git workspace snapshot.MEMORY.md, USER.md, plugin sections, profile line, timestamp line, runtime environment hints (host / home / cwd).Stable content leads. A timestamp that changes every turn sits last — after it, nothing needs to survive.
Some are physics. Some are provider policy. Some are economics. None can be tuned away — they can only be designed around. Tap a card for the evidence in the source.
Anthropic permits a maximum of four cache_control breakpoints per request. You cannot mark every meaningful boundary — so you spend the budget where reuse is highest. Hermes' default spends two on the system prompt and rolls the last two across recent messages.
stable tier — byte-identical, cached across every turn of the conversation.tool_result.content[0] → a non-retryable HTTP 400. There, tool parts carry no marker at all.Entries live for 5 minutes or 1 hour. A write costs a premium (1.25× for 5m, 2× for 1h); a read costs ~0.10×. If the next turn arrives after expiry, you pay a cold re-write at full cost. So the optimal TTL is a function of how far apart your turns arrive — nothing else.
1h for human-typed sources (cli, tui, desktop, messaging). 5m for machine-paced sources: subagent, cron, oneshot, webhook, kanban, api, tool, batch. Auxiliary calls treat auto as 5m.A write's premium is only recovered by a later read. Model a fixed 100K-token prefix reused across N turns: the no-cache line is linear; the cached lines carry an upfront premium then run at a tenth of the cost. Where they cross is the break-even — ~1.3 turns for the 5m tier, ~2.1 for the 1h tier. A single-shot request never profits.
/model switch, a fallback, a credential-pool rotation onto another account — means the next request gets zero cache hits and re-reads the whole conversation at undiscounted price. Inherent to the mechanism./new starts a fresh scope; forks stay isolated.The prompt architecture is an accumulation — each block carries a provenance comment naming where it came from. Tracing them is tracing the intellectual ancestry of the modern agent harness.
Cache efficiency is one room in a large house. Everything above depends on the engine underneath it — a single conversation loop that drives every turn, and the subsystems hung off it. Here is the rest of the machine.
run_conversation()step 1 / 6The loop runs the same shape for every turn: build, call, act, append, repeat — and compress when the window fills.
Tap any card for what it is, how it works, and where it lives in the source.
tools/your_tool.py with a top-level registry.register() call, add the name to toolsets.py, and the auto-discovery imports it. Every handler returns a JSON string. A check_fn hides the tool until its requirements are met.SKILL.md with YAML frontmatter (name, description, triggers) plus optional references/, templates/, scripts/. The agent loads them on demand — the index is in the prompt, the body is fetched only when used. Skills are the agent's procedural memory.Drawn from a shallow clone of NousResearch/hermes-agent at commit b2360086. Primary evidence:
Cache-cost multipliers (0.10× read · 1.25× / 2.00× write) are vendor-tier reference values used to model the economics. The break-even conclusions follow directly from them and are independent of any single provider's exact rate card.