hermes-agent architecture: caching outranks compaction

Two settings in hermes-agent’s config decide how it shrinks a conversation that has outgrown its context window. Both ship at zero. The expensive one, an LLM that summarizes your history and hopes the summary was faithful, is the only one switched on.

That inverts the advice I would have given, and have given: reduce cheaply and reversibly first, summarize last if at all.

The comment beside each zero gives the same one-line reason, and it is the key to the hermes-agent architecture. A prune rewrites history the provider has already cached.

The shape, from a height

NousResearch’s hermes-agent is a personal assistant you reach from a terminal, a desktop app, a code editor, or any of about twenty chat platforms, and it is the largest open-source harness I have read. It is also a clean example of the runtime layer I usually end up writing about: almost nothing here is about the model.

✕

One core sits behind all of it, and the interesting thing is what that core refuses to do. Two things never change mid-conversation: the system prompt and the tool schema. Both sit at the front of every request.

A turn then goes like this. The core assembles the request, a planner marks four points in it where the provider should save its work, the marked-up copy goes out, and the provider replies with a count of how many prompt tokens it actually billed. That number, not the harness’s own guess, decides whether any of the reduction paths at the bottom of the diagram fires.

Everything below is downstream of one arithmetic fact about that last number, so it is worth doing the arithmetic before looking at any of the machinery.

Why a cache is worth protecting this hard

Providers charge a reduced rate for the part of a request they have already processed, on the condition that the bytes match exactly from the very first byte. Their own pricing table lists one current model at five dollars per million input tokens, fifty cents per million for those already-processed reads, and six and a quarter to save them in the first place.

I read this code rather than running it, so treat the rest as arithmetic on their figures.

A re-read costs a tenth of a fresh read. Saving costs a quarter more than not saving. So a prefix you save and never read again is a loss, and one you read back even once has already paid for itself.

Which reframes what a cache miss is. Breaking the prefix is not losing a discount. It is paying a penalty up front, then paying full price to re-send everything behind the break.

On a three-turn task none of this matters. On a conversation that runs all day, the front of the request is re-sent a hundred times, and it becomes the largest recurring line on the bill. That is the workload this harness assumes, and it is stated as one of two properties the contributor guide says every change is reviewed against:

What that contract buys, and what it costs

Once a rewrite is a penalty, the cheap reducers stop being cheap.

The deterministic prune is the pass I would put first in any other harness. It walks old tool results, drops duplicates, summarizes oversized ones without a model, truncates bulky arguments, and leaves the full output on disk. Predictable, reversible, no model call. It ships at zero.

Micro-compaction, which folds the oldest exchange into a rolling summary after each turn, is off for the same reason.

So the shipped ladder is: nothing, then nothing, then one large summarization at three-quarters of the usable input budget.

The choice	What it buys	What it costs
Prune on, the usual advice	Dead tool-result tokens stop being re-sent	A re-ingest of everything behind the edit, every time it commits
Prune off, carry the junk	The prefix stays warm, so the bill stays at a tenth	Window: dead tokens pull the expensive summarization forward
Micro-compaction on	Occupancy stays flat, and on their published run the big summarization never fired at all	A broken prefix every turn, plus a pass that ran two to thirty-seven seconds each time

I went in expecting to file those two zeros as an oversight, a half-built feature someone left switched off. The zero is the decision, and the feature exists for the workloads where the arithmetic tips the other way, which is most of them.

Five minutes

There is a hole in all of this, and I re-read the config twice before I noticed it.

A cached prefix has a lifetime. The shipped default here is the five-minute tier.

Now put that against the workload the architecture is justified by: terminal in the morning, phone at lunch, a cron job overnight, all one conversation. At five minutes, the prefix is stone cold at every one of those transitions, and the first turn back pays a full write no matter what anyone did to the transcript. Nothing in the cache planner looks at how long the session has been idle.

Which means the coldest moment in this workload is also the cheapest moment to rewrite history, because there is nothing left to invalidate. A prune that would be expensive mid-burst is free on resume.

They built the machinery for exactly that moment. Idle compaction fires when a session resumes after a configurable gap and reduces the transcript before the first reply. It is opt-in, it ships at zero like the others, and it calls the summarizer rather than the cheap deterministic pass.

So the rule this architecture implies, and stops one step short of adopting: prune hard on resume, never mid-burst.

The mechanism I would copy tomorrow

Everything above hangs off one number: an estimate of how large the next request is. Every harness computes that locally, before sending. Every harness gets it wrong, because tokenization depends on content you are guessing about.

Hermes treats that error as a measurable quantity.

Their estimator over-counts on purpose, so compaction happens before a provider rejects the payload. But the margin is not a fixed percentage. Their own note puts CJK text at about seventy percent over its real cost, and reasoning-replay blobs at several times theirs. A heavy session can show an estimate two or three times real usage.

Sit with what that means. You believe you are at ninety percent of the window. You are at forty. So you pay for a summarizer call you did not need, stall a turn nobody needed stalled, and throw away detail for nothing, and none of it shows up as an error.

So they close the loop:

before the call:  rough = estimate(messages, tools)
after the call:   real  = response.usage.prompt_tokens
                  if the prompt fit: baseline = (rough, real)

# fire on the projection, not the raw guess
growth    = max(0, rough_now - baseline.rough)
projected = baseline.real + growth
if projected >= threshold: compact()

Two things make this better than it looks. The pair is re-anchored on every response whose prompt fit, not just after a compaction, so the correction tracks the session as it drifts. And the harness is candid about where the correction fails: it is not a strict upper bound for Cyrillic, Greek, Thai or Arabic, with a provider overflow error as the backstop.

Notice what this buys, because it is unusual. A trigger that fires at the right time is more accurate, since detail stops being discarded for no reason. It is cheaper, since the summarizer call is not made. And it is faster, since the stall does not happen. Most context work buys one of those by giving up another. This one gives up nothing to get all three, which is rare enough to be worth stopping on.

Then it does the half nobody else does.

✕

The check most harnesses run after compaction is “is the message list shorter?” That check can lie, because the thing you shortened and the thing you are billed for are not the same object. Here is what actually goes out every turn:

[ system prompt ][ tool schemas ][ ...messages... ][ new turn ]
└── the floor: 20-30K tokens ───┘└── compaction ──┘
   their figure, sent in full every turn      the only part a pass can shrink

Three ways a shorter list is not a smaller bill. The floor can sit above the threshold on its own, in which case every pass shrinks messages by a healthy margin and the prompt stays over the line forever, which is their comment almost verbatim. Your local estimate can be wrong in either direction, so a post-compaction reading of “still at 80%” may be a real 55%. And inserting a summary is not free: the marker scaffolding alone runs about four hundred tokens, and their own published run shows a first pass that added tokens rather than removing them.

So the verdict is judged on the provider’s billed count, which is the same number the whole exercise exists to reduce. Close the loop on the signal you are optimising, not on a proxy you compute yourself.

That verdict necessarily arrives late. You cannot know the bill until the next request completes, so the sequence is compact, send, read the bill, then judge the compaction you already committed to. Which is why the mechanism needs a memory.

A reading still over the line records a strike. Two strikes stop automatic compaction, and so does a summarizer that keeps falling back to deterministic dropping, since the first says reduction is not helping and the second says the summarizer is unhealthy. The code does not say why two rather than one; the obvious reading is that a single bad reading can be timing, a large paste landing right after a pass, while twice in one session is structural.

The counters live on the session row rather than in memory, and that is the quietly clever part. Picture the in-memory version: a session gets stuck compacting to no effect, it feels slow, so someone restarts it, and the counter resets to zero and the loop starts again. The situations where you most need the breaker are the ones most likely to produce a restart. Every surface on the machine shares that one store, so the failure history belongs to the conversation rather than to the process.

Nor is the stop permanent, which took a shipped bug to get right. A guard was tripped against the transcript as it looked at that moment, and conversations keep growing, so after five minutes of continuous block one probation probe is allowed. Fail it and it re-trips on the very next verdict, which bounds a genuinely incompressible session at one attempt per window.

Be exact about what an open breaker costs you, because it is not free. Automatic compaction stops, the prompt keeps growing, and each turn is therefore dearer than the last, with a provider overflow error as the eventual backstop. What you stop paying for is the summarizer calls and the stalls, which were buying nothing. Read the breaker as a diagnosis rather than a repair: your floor is too big, and no schedule of summaries fixes a floor.

Measure honestly, tolerate noise, fail safely and durably. That is the shape I would want around any expensive automated action that can quietly stop working.

What I cannot tell from reading is how much the correction buys on a typical session, since nothing in the repo reports corrected triggers against raw ones. The mechanism is clearly right; the size of the win is their figure rather than mine.

Three more decisions worth appreciating

Only the builder knows where a message’s cacheable part ends. Skill, webhook and cron messages are a large static scaffold concatenated with a small volatile tail. Cache the whole thing and you pay a fresh write every invocation. So the code that built the message registers the byte where the scaffold ends, and the planner reads that registry rather than guessing. They refuse to search for a delimiter, because a delimiter can legitimately appear inside a scaffold, and a bad guess silently absorbs volatile bytes into the cached prefix, reintroducing the exact miss it was meant to remove. Cheaper, with no accuracy risk at all.

The approval bypass is frozen when the code loads. The agent runs skills and generated code inside its own process, so if the bypass flag were read from the environment on every check, one line of injected Python would disable every approval for the rest of the process, with no prompt and no log. Reading it once at import turns a runtime switch into a launch-time argument that model-generated code cannot reach. Control, at zero runtime cost, and the sort of thing you only think of after imagining the attack.

Compaction is a flag flip, not a delete. The archived messages stay on disk, still searchable, marked in a way that distinguishes “summarized away” from “the user rewound it”. And because the search index only fires on content columns, flipping those flags does no index work at all, so archiving ten thousand messages is one cheap update. Durability and speed at the same time, which is what a good schema choice looks like.

There is a fourth I will mention in passing because it is the clearest latency win in the codebase: when the model emits several tool calls at once, the batch is cut into runs that can safely overlap, reserving file paths as readers or writers. Concurrent reads of the same subtree commute; a writer conflicts with any overlap. So independent reads run together while the read-after-write ordering the model depended on is preserved.

Where the design runs out

Every number above is measured against a threshold, and the threshold has to clear that floor: system prompt plus tool schemas.

Their reasoning for its size is good and written down. The core is described as a narrow waist with capability at the edges, and the stated reason is the right one, that every tool they add is sent on every call. A ladder pushes new capability toward a CLI command plus a skill, then service-gated tools, then plugins, with a core tool as the last resort. Most projects have nothing like it, and the always-available surface is the product.

And yet the core list runs to fifty-nine tools, and their own comment budgets the resulting floor at twenty to thirty thousand tokens. Under this harness’s own economics that floor is cheap in money, since it sits ahead of every rewrite point and is re-read at a tenth of the price. What it costs is window, and therefore time: it is why the compaction trigger has to be raised from half to three-quarters on models under a 512K window, and their comment says what happens otherwise, which is that compaction re-fires every turn or two and the session spends its wall clock summarizing.

So what does this harness optimize for? Not input-token count, which it will let grow rather than rewrite history. It optimizes for the turn economics of a long-lived conversation, and specifically for the cached prefix. Not cost per successful task either, though the calibration loop and the breaker come closer than most, since both exist to stop it spending money on work that produces nothing. The gap is precise: they catch wasted harness work rather than wasted task work. Nothing here notices that the agent got the answer wrong and is about to retry the same failing approach. The strikes-and-breaker pattern would port to that problem unchanged, two identical failing tool calls and stop, and it is not applied there. It audits the plumbing, not the plumber.

The one that will bite you

Cache markers have to be applied dead last, after every other change to the transcript, and nothing in a type system will tell you when you have broken that.

Marking a message rewrites its content from a plain string into a list of blocks. A later pass that type-checks for a string then skips it. So a tool result ending in a newline ships unstripped while it sits in the marked window, and stripped once it rolls out: same message, different bytes on consecutive turns, breaking the prefix exactly where the markers existed to protect it. You find out weeks later from a sag in your cache hit rate, with no error anywhere.

This harness gets the order right and explains it in a comment. If you build a cache plan, make it an assertion.

Where this leaves us

The rule I am keeping is narrower than the one I arrived with. Cheap context reduction never touches bytes the provider has already seen, and the next cheapest happens at a boundary you were already paying for: a compaction commit, or a cache expiry.

What stays open is where the rewriting happens at all. This harness already archives rather than deletes, so the durable record is append-only in the sense that matters. What gets rewritten is the outgoing request, and that is what all the gates and cooldowns and durable counters exist to protect. So why is the request assembled by editing history, rather than projected from it, so what the model sees can change without changing a byte of what the provider has seen?

If you want the rest of it, the loop and its budget, the tool layer, state, approvals, delegation and providers, that is the component-by-component study.

I’m Kaushal Prajapati, a Staff AI/ML Engineer. More posts, or say hi on LinkedIn.