hermes-agent internals, one component at a time
A component-by-component study of hermes-agent internals: the loop, the prompt, the cache plan, compaction, state, approvals, and what each one costs.
An agent harness is the code between a language model and the world. It holds the conversation, decides what the model is allowed to call, runs those calls, and decides what the model gets to see on the next turn. Most of what makes an agent feel competent or useless lives in that code rather than in the model.
So this is a walk through one, end to end. These are the hermes-agent internals, which belong to NousResearch and make up the largest open-source harness I have read. hermes-agent is a personal assistant you talk to from a terminal, a desktop app, a code editor, or any of about twenty chat platforms. We start with the loop, which is about eight lines, then add the scaffolding around it in the order the design forces: the prompt, the tool schema, the cache plan, the three ways it shrinks a conversation, where state lives, how dangerous actions get approved, and what happens when it hands work to a child.
This is the long version, and it is built to be walked or skimmed: every chapter below stands on its own, so the headings are a menu rather than a sequence. If you want the architecture at a glance and the handful of things that make it unusual, read the short one instead.
Two things are worth knowing before we begin. It treats one provider feature, the prompt cache, as a constraint that outranks its own best features. And it is the only harness I have read that goes back afterwards and checks whether its own context reduction accomplished anything.
I read this code; I did not run it. Where I put a number to something, it is arithmetic on the repo’s own published figures.
Start with the loop, because everything else is scaffolding
Every agent harness has this at its centre, and it is worth writing out because the rest of the post is a response to it.
# the whole idea, in eight lines
messages = [system_prompt, user_message]
while True:
reply = model(messages, tools=tool_schemas)
messages.append(reply)
if not reply.tool_calls:
return reply # the model has stopped asking for things
for call in reply.tool_calls:
messages.append(run(call)) # this is where the world happensNow look at what is wrong with it, because each fault is a component later in this post.
One. The message list grows every iteration and never shrinks, so eventually the request does not fit, or costs more than the task is worth. Two. The loop needs a stop condition, and the model cannot be trusted to provide one, because a confused agent will keep calling tools forever. Three. The tool schema is re-sent on every single iteration, so a large toolbox is a tax on every turn rather than a one-time cost. Four. Running a call will happily delete a directory if the model asks it to.
Four faults, four subsystems, and we will take them in that order. Hermes-agent’s answers to them are more interesting than most, and they are all downstream of one decision about what kind of conversation it is having.
The workload that explains every other choice
You talk to this thing from your terminal in the morning, your phone at lunch, and a cron job runs it overnight, and the design assumes that is all one conversation, hundreds of tool results deep.
That assumption is doing a lot of work, and it is a runtime-layer question rather than a modeling one, which is the layer I usually end up writing about. So it is worth being concrete about the consequence. Providers will charge you a reduced rate for the part of a request they have already processed, on the condition that the bytes match exactly, starting from the very first byte. Their own pricing table lists one current model at $5.00 per million input tokens, $0.50 per million for those already-processed reads, and $6.25 to save them in the first place.
Read those three numbers again, because the middle one is the whole architecture. A re-read costs a tenth of a fresh read. But saving costs a quarter more than not saving, so a prefix you save and never read again is a loss, and one you read back even once has already paid for itself.
On a three-turn task this barely matters. On a conversation that runs all day, the front of the request gets re-sent a hundred times, and the difference between paying a tenth and paying full price for it becomes the largest recurring line on the bill. Which is why the contributor guide opens with this, as one of two properties they say every change is reviewed against:
Keep that sentence in mind. Almost every design choice below is either an application of it or a workaround for it.
The map
So here is the whole harness at block level, and we will visit each box in turn.
Six surfaces share one core. Inside that core are the four things this post is about: the loop and its budget, the registry that decides which tools exist this turn, the system prompt, and the planner that marks up the request for caching. Below it sit the three subsystems that intervene when something goes wrong, context grows too large, an action is dangerous, a task is too big for one agent, and underneath everything, the store that remembers what happened and the provider that tells you what it cost.
The system prompt is built in three tiers, and the order is the point
The prompt is assembled once per session and reused on every turn afterwards; only a compaction triggers a rebuild. It is joined from three tiers, in this order:
- stable, the agent’s identity, tool guidance, per-model operational notes, environment hints.
- context, anything the caller passed in, plus context files like AGENTS.md discovered under the working directory.
- volatile, the skills index, a memory snapshot, the user profile, and a line naming the session, model and provider.
The tiers exist entirely for the cache. Since a provider only reuses work when the bytes match from the beginning, you put the things that never change at the front and herd everything that might change into a tail you have already given up on.
Notice the small detail in the volatile tier: the date line records when the conversation started, rather than what time it is now. A clock in a prompt would change the bytes on every turn and cost you the entire prefix. This is the kind of thing that is obvious in hindsight and invisible until it costs you money.
Four breakpoints, and the discipline of not spending one
A cache breakpoint is how you tell the provider where to save its work, and you get four per request. That is the entire budget, and everything interesting here is in the allocation.
The default layout spends one on the end of the stable tier, which survives everything, one on the end of the system prompt, which survives until a compaction, and the remaining two on the last two cacheable messages, which move forward every turn. When there is no static prefix to split, it falls back to one on the system prompt and three on the recent tail.
Two on the prompt, two riding at the end of the conversation. The two at the end are the working pair: each turn spends one save on the messages it just added, and every later turn reads them back at a tenth of the price. The two at the front are the insurance, and they are why a new session on the same machine can start warm.
Three refusals separate this from stamping markers on the last few messages.
It will not spend a breakpoint on a message the provider would ignore. On envelope-style routes, a marker attached to an empty assistant turn, one that is nothing but tool calls, gets silently dropped. A predicate filters those out first, so the four land on messages that actually count.
It will not put a breakpoint in the middle of a tool run. An assistant message with three tool calls, followed by three results, is one transaction; only the last result is offered as a legal boundary. Cut in the middle and the prefix you cached ends at a point the next request will not reproduce.
It will not ship a marker the route does not honour. A configured one-hour cache tier clamps back to five minutes on Qwen and Alibaba routes, because those honour a five-minute window and ignore the longer tier. Sending it anyway would cost the write premium and create, in their words, a false expectation.
The whole plan is applied to a deep copy of the messages. What gets stored on disk stays a plain transcript with no cache markup in it, which means a provider that does not support caching sees a clean request and the persistence layer never has to know any of this exists.
The failure they designed for: what happens after a restart
The exact bytes of that stable tier are never saved to disk. Only the full prompt is. So a process that restarts into a stored session has no record of where the cacheable part ended.
It rebuilds the stable tier from scratch and then checks whether the stored prompt literally starts with it. If it does, the boundary is recovered and caching continues. If anything feeding that tier has changed since the prompt was written, someone edited the identity file, the rebuilt prefix will not match, and rather than patch the difference, the harness gives up on the optimization and sends the stored bytes untouched.
Name that as a rule: when you cannot prove where a cache boundary is, lose the discount rather than rewrite the bytes. A wrong boundary does not merely miss the cache, it silently poisons every later turn.
When only the builder knows where the prefix ends
One mechanism here solves a problem you only hit at a certain scale.
Some messages are machine-built: a skill invocation, a webhook delivery, a cron job. Those get assembled by concatenating a large static scaffold, the expanded skill body, the activation notes, with a small volatile tail, like the ticket payload or the timestamp of this particular run. The scaffold is identical every time. The tail never is.
If you cache that message as one block, you pay a fresh write on every invocation, because the tail moved. If you could split it at the byte where the scaffold ends, the expensive half would be cached and only the cheap half would be new.
You might be thinking: just find the delimiter and split there. So was I, and the module rejects that explicitly. A scaffold marker can legitimately appear inside a skill body or inside an event payload, and a delimiter search that guesses wrong either shrinks the cached region or, worse, absorbs volatile bytes into the cached prefix, which reintroduces the per-invocation miss you were trying to eliminate, silently.
So the code that built the message registers where its stable prefix ends, at construction time, and the planner consults that registry instead of guessing.
# at construction time, the builder knows the offset
register_stable_prefix(scaffold) # skills, webhooks, cron all do this
# at request time, the planner looks it up rather than guessing
prefix = find_stable_prefix(message)
if prefix:
split message into [cached: prefix] + [uncached: the rest]The registry is bounded two ways, at thirty-two entries and a few megabytes, evicting least-recently-used but never the newest entry, so one oversized scaffold cannot silently disable the split for everyone else. A cron job firing every minute also refreshes on use, so it does not get evicted by a burst of one-off skill calls.
I have not seen another harness do this. It is a clean example of a general principle: the component that assembled a thing knows something about it that no later inspection can recover, so have it declare that knowledge rather than re-derive it.
Three ways to shrink a conversation, and two of them ship switched off
That prompt sits in front of a transcript that grows forever, which was the first fault in our eight-line loop. Hermes has three answers to it, and this is where I stopped and re-read the config file twice.
The deterministic prune walks old tool results, drops duplicates, summarizes oversized ones without calling a model, truncates bulky call arguments, and leaves the full output retrievable from the store. Cheap, predictable, reversible. Its trigger ships at zero.
Micro-compaction folds the oldest un-absorbed exchange into a rolling summary after each turn, so occupancy stays flat instead of sawtoothing up to a threshold. It ships off.
Batch compaction waits until the prompt crosses a threshold, then calls a model to summarize the middle of the conversation. It is the expensive one, the lossy one, and it is the only one enabled by default.
If you have read anything about context management, that ordering is backwards. The standard ladder caps what you write, evicts what is stale, and summarizes only as a last resort, because summarizing costs a model call and loses detail you cannot recover. Hermes ships the last rung and leaves the first two to you.
The comment beside each disabled setting gives the same reason, and by now you can predict it: each committed prune rewrites already-sent history, breaking the provider prompt-cache prefix.
That is the trap in the standard advice. Eviction is free when the deleted tokens are the only thing you were paying for. In a cached conversation, deleting a message from the middle re-prices every message behind it, because the byte-exact match the discount depends on ends at your edit.
| The choice | What it saves | What it costs |
|---|---|---|
| Prune on, the usual advice | The dead tool-result tokens on every later turn in the burst | A re-ingest of everything behind the edit, every time it commits |
| Prune off, carry the junk | Nothing. The junk rides along in history | Cached re-sends at a tenth of fresh input, plus window: dead tokens pull the expensive summarization forward |
| Micro-compaction on | On their published run, occupancy flattened at about a fifth of the threshold and batch compaction never fired at all | A broken prefix every turn, and a pass that ran two to thirty-seven seconds, median around thirty-one, on one contended local model |
Those columns are honest about money and vague about accuracy. Carrying junk is cheap in tokens, but stale tool output is not inert: a superseded file listing is something the model can read and act on. Nothing in the repo measures that, and neither can I from reading it.
I went in expecting to file those two zeros as an oversight, a half-built feature left switched off. The zero is the decision, and the feature exists for the workloads where the arithmetic tips the other way.
What happens when you do switch pruning on
The config comment names the workload where you should: a large-window model where the threshold rarely fires, so bulky terminal dumps and file reads ride along in history and get re-sent every turn.
A prune only commits if it reclaims about four thousand tokens, roughly the weight of two oversized tool results by the repo’s own definition. Why refuse a smaller win? Because a prune that fires whenever it can fires on nearly every iteration: each new tool pair ages an older one out of the protected tail, so there is always something to drop, and every one of those fires would break the cache. The gate insists on a meaningful batch before paying that price.
Then it adds hysteresis, which is the part I would not have thought of.
# a prune commits only if it clears the gate
reclaimed = before - after
if reclaimed < MIN_RECLAIM: # a few thousand tokens
keep the junk, try again later
# and then it must wait out a runway before re-arming
runway = max(reclaimed, trigger, MIN_RECLAIM)
rearm_at = after + runway # written to disk with the compacted rowsAfter a commit, the next prune is blocked until the prompt regrows at least as much as was just reclaimed, and that watermark is written to disk alongside the compacted rows. That last part is the detail worth stealing: someone worked out that restarting the process is the obvious way to defeat a cooldown held in memory, and closed it.
Hysteresis is not free. Junk now lives in context for up to a full runway longer than it needs to, and a batched schedule drops a different set of messages than an eager one would. Against a per-turn cache break it is still cheaper.
The number that fires all of this is wrong
Every gate above is downstream of one quantity: an estimate of how large the next request will be. Every harness computes that locally, before sending, and every harness’s estimate is wrong, because tokenization depends on content the harness is guessing about. Hermes is unusual in treating the error as measurable rather than as a rounding allowance.
Their estimator over-counts on purpose, so that compaction happens before a provider rejects the payload, but the margin is not a fixed percentage. Their own note puts CJK text at about seventy percent over its real cost and reasoning-replay blobs at several times theirs, so a heavy session can show an estimate two to three times real usage and fire compaction at a third to a half of the real window.
Sit with what that means. You believe you are at ninety percent of the window; you are actually at forty. So you pay for a summarizer call you did not need, stall a turn nobody needed stalled, and throw away detail for nothing. The bug is invisible, because everything downstream looks like it worked.
So they close the loop. The rough estimate is recorded before each call, paired with the provider’s billed prompt-token count when the response arrives, and re-anchored on every response whose prompt fit rather than only after a compaction. The trigger then fires on a projection rather than the raw guess.
before the call: rough = estimate(messages, tools)
after the call: real = response.usage.prompt_tokens
if the prompt fit: baseline = (rough, real)
# fire on the projection, not the raw estimate
growth = max(0, rough_now - baseline.rough)
projected = baseline.real + growth
if projected >= threshold: compact()Look at the shape of that and you can find the blind spot yourself. It corrects the bias it has already observed, then trusts the new increment at face value, so what it cannot anticipate is a fresh paste of exactly the content the correction exists for. The module is candid: the projection is not a strict upper bound for Cyrillic, Greek, Thai or Arabic, and it names two backstops, a real reading at or above the threshold clearing the baseline, and a provider overflow error still triggering compaction.
One thing I could not work out from reading: how much the correction actually buys on a typical session, since nothing in the repo reports corrected against raw trigger points. The mechanism is clearly right; the size of the win is their figure rather than mine.
And then it asks whether the compaction was worth doing
This is the half I had not seen in another harness, and it is why the diagram above has a second row.
A calibrated trigger tells you when to compact. It says nothing about whether compacting helped. So after the next response arrives, the harness compares the provider’s billed count against the threshold and asks whether the prompt actually got under the line.
Note what it is not asking: whether the message list got shorter. Those two come apart in three distinct ways. The floor can exceed the threshold on its own, the system prompt plus the tool schemas that every request carries in full. A local estimate can be wrong in either direction, so “still at 80% after compacting” may be a real 55%. And inserting a summary is not free: the marker scaffolding alone runs about four hundred tokens, and their published run shows a first pass that added tokens rather than removing them. Their comment is the sentence I would put in front of anyone debugging a compaction loop: when that floor alone meets the threshold, “every pass shrinks messages by a healthy margin yet leaves the prompt over the line, so the next turn compacts again, forever.”
The detail that makes this work is where the check lives, and the comment is explicit that it must not live in the obvious place.
It would be natural to put it in the function that decides whether to compact. That function runs twice per turn, on two different measures: a rough preflight estimate before the request, and the provider’s real count after it. And the rough one can dip below the threshold on its own, which would clear the strike and reopen the loop every single turn. So the verdict is judged only where the real post-response count is visible, which is the same place the calibration pair is re-anchored. Like compared with like, exactly once per compaction.
Once per compaction is enforced by a latch. Committing a boundary arms it; adjudicating clears it. And if the next response arrives with no usage block at all, the latch is consumed anyway rather than left standing, so a reading that lands three turns later cannot be charged to a compaction it has nothing to do with.
# one boundary, exactly one verdict
after a compaction commits: verify_next_real_reading = True
on a provider response:
if it reports no usage: verify_next_real_reading = False # not this one's verdict
elif verify_next_real_reading:
strikes = strikes + 1 if billed_prompt_tokens >= threshold else 0
verify_next_real_reading = False
# checked before every compaction
if strikes >= 2 or fallback_streak >= 2:
if no deadline yet: deadline = now + 300s # armed on first BLOCK, not at trip
elif now >= deadline: drop tripped counters to 1 # one probation probe
else: refuse to compactNote the second condition on that guard. Two things can trip it independently: two ineffective verdicts, meaning compaction ran and the prompt stayed over the line, or two consecutive boundaries that fell back to deterministic dropping instead of a model summary. The first says reduction is not helping. The second says the summarizer itself is unhealthy. Both end in the same place, and keeping them as separate counters is what lets the log tell you which failure you have.
There is a third case that deliberately counts as neither. When the compressible middle is too small to be worth a model call, the pass skips the summarizer and drops deterministically. That skip must not extend the fallback streak, because two skips would trip the breaker and disable compaction entirely, including the cheap deterministic dropping the skip existed to reach. It must not reset the streak either, since skipping proves nothing about the summarizer’s health. So it is streak-neutral, and yet it still arms the effectiveness verdict on purpose: a skipped-summary drop that fails to clear the threshold is precisely the incompressible transcript the breaker exists to catch.
Then there is the question of how the block ends, which the comment treats as a bug they had already shipped. A tripped guard was judged against the transcript as it existed at that moment, when the middle may have been too small to matter. Conversations keep growing. Without a way out, a session never auto-compacts again and rides into the provider’s hard context limit.
So the recovery is a probation probe. After five minutes of continuous block, the tripped counters drop to one strike and a single attempt is allowed. If that probe also fails to clear the threshold, the very next verdict re-trips the guard, which bounds the worst case in a genuinely incompressible session at one compaction attempt per recovery window. The drop is persisted, so sibling agents bound to the same session row unblock together rather than each discovering it separately.
The clock is the part I would copy verbatim. The deadline is armed on the first blocked evaluation rather than at trip time, and measured on a monotonic clock. A fresh process that loads a durable tripped counter therefore starts a full window blocked instead of finding an expired deadline and immediately probing. Restarting cannot disarm the guard, which is the whole point of persisting it.
Where it lives: two integer columns on the session row, both defaulting to zero, alongside the summary-failure cooldown. One bind call rehydrates all of them when a compressor attaches to a session.
Be clear about what tripping buys and what it leaves. It ends the spending on passes that reclaim nothing. It leaves a session that can only grow toward the provider’s hard limit. Read the breaker as a diagnosis rather than a repair: your floor is too big, and no schedule of summaries fixes a floor.
The boundary they built and left switched off
One thing about caching gets skipped in almost everything written about it: a cached prefix has a lifetime. The shipped default here is the five-minute tier, with an hour available on request.
Now put that next to the workload. Terminal in the morning, phone at lunch, cron overnight. At five minutes, the prefix all this machinery protects is stone cold at every one of those transitions, and the first turn back pays a full write no matter what anyone did to the transcript. The contract binds hard inside a burst of turns and not at all across the gaps between them, and nothing in the cache planner looks at how long a session has been idle.
Which points somewhere the defaults do not go. A prune is expensive when the prefix is warm and free when it is cold, and the coldest moment in this workload is the one the product is built around: you come back after lunch.
Hermes has machinery for exactly that. Idle compaction, opt-in and shipping at zero like the others, fires when a session resumes after a configurable gap, above a size floor, and pays for a full summarization before the first reply. That is the expensive reclaimer arriving at the cheap boundary, with the cheap one still wired to nothing. The code justifies it as not re-reading stale context and never says the quiet part: a resume past the cache lifetime is the one moment when a rewrite costs nothing, because there is nothing left to invalidate.
Prune hard on resume, never mid-burst, is the rule this architecture implies and stops one step short of adopting.
The second fault: knowing when to stop
That is the first fault handled. The second was the stop condition, and it lives in the loop’s own header. Here is the real condition:
while (api_calls < max_iterations and budget.remaining > 0) or grace_call:
reply = model(messages, tools=snapshot)
api_calls += 1
budget.consume()
...run tool calls, append results, continue...Three things in that line are worth knowing, and two of them are archaeology.
The ceiling first. The iteration cap is how many times the model may be called before the turn is cut off, and finding its default is a small adventure: the constructor says ninety, the CLI hands it five hundred, a subagent gets two hundred and fifty from config, and the budget class’s own docstring says five hundred for parents and fifty for children. Four numbers, and only one of them is what you get. Read the code, not the comments, including in this post.
The second guard, the one about remaining budget, looks like a separate allowance and is not. The budget object is thrown away and rebuilt at the start of every turn, and both counters move together, so the two halves are the same predicate. The object exists for a case where a shared budget is injected from outside; the turn prologue overwrites it before it can matter.
And the third clause is dead. That grace flag is initialized to false, read twice, and never set to true anywhere in the repository. The behaviour it describes is real, but it moved: when a turn ends without the model producing any text, the finalizer makes one extra call with the tool schema stripped out entirely, asking for a summary of what happened. Stripping the tools is the part worth copying, because “please stop calling tools” as an instruction is a request, and removing them from the schema is a fact.
Why a counter that goes backwards is the interesting part
Six places in that loop refund an iteration, decrementing both counters together.
The rule behind it: the budget measures model turns the user asked for, not attempts the machinery made. So an iteration that never reached the provider does not count: a compression pass that fired before the request, a local runtime whose context turned out too small to hold the tool schema, a mid-turn correction that cancelled the in-flight call, a provider failover.
That last one is a quality decision rather than bookkeeping, and the comment says so, the refund exists so the fallback provider “gets a fair turn” instead of inheriting a budget already spent by the provider that failed.
The sixth refund is the one I would steal. If the only tool the model called in an iteration was the code-execution tool, the one that lets it batch many operations into a single programmatic call, the iteration is free. That is a price signal aimed at the model: the cheaper pattern costs it nothing from its budget.
The cost of all this is six hand-maintained decrement pairs, and the comments record what happens when one is missed. A skipped turn once leaked a budget unit for the life of the agent, and the turn log reported an API call that never happened.
Worth separating two jobs that look alike here. A counter that can go backwards is an accounting mechanism. The things that actually stop runaway behaviour in this codebase are elsewhere: retry ceilings, cooldowns, and the breakers we met earlier.
Interrupting a turn without breaking the cache
There is a feature in that loop I did not expect: you can talk to the agent while it is working, and your correction lands mid-turn.
That sounds simple and is not, because a turn in progress holds provider-specific state. Reasoning blocks from a thinking model are signed, or require the output that follows them, so you cannot staple a user message onto a half-finished assistant turn and expect the next request to validate.
What it does instead: keep only the visible response text produced so far, demote it to ordinary assistant text, then append your correction as a real user message. Role alternation stays valid, the provider gets a replayable transcript, and, in the code’s own words, every previously cached message is left byte-for-byte unchanged. Even the interrupt path protects the prefix.
There is an invariant next to it worth adopting wholesale: raw chain-of-thought must never be serialized into replayable message content. Streamed reasoning is display state. Show it live, but do not let it re-enter the transcript, or the next request reads as though the model said all of its thinking out loud.
Where the conversation actually lives
One SQLite file, in write-ahead mode so many readers can coexist with a single writer, with a full-text index over every message. The write-ahead mode is not a detail: a gateway serving twenty chat platforms, a desktop app, a CLI and cron are all reading that file at once.
The full-text index is not a convenience for a history browser, either. It is the agent’s long-term recall: a search tool runs FTS5 over every message you have ever exchanged, dedupes hits by conversation, and returns snippets with a window of messages around each one, with no model calls anywhere in the path. Recall, in this design, is a database query rather than an embedding lookup.
One small thing in the load path travels anywhere. The conversation load orders messages by their auto-incrementing row id, never by timestamp, and the comment explains why: time.time() is not monotonic. Sleep a laptop, step a clock with NTP, run under WSL, and a row written later can carry an earlier timestamp than the row before it. Sort by time and you can put an assistant’s tool call after the tool’s response, which is not a cosmetic problem. It is a malformed request the provider rejects.
Compaction is a flag flip, not a delete
Now the question this chapter exists for. If compaction throws away messages so the model can keep going, how do you ever go back and see what actually happened?
You do not throw them away. Compaction runs as one transaction that flips every live row to inactive, marks it as compacted, and inserts the summary and the surviving tail as new live rows. The session keeps the same id for its whole life. Nothing is deleted.
one write transaction:
mark every live row active=0, compacted=1 # the old set, archived
insert the new set active=1 # summary + surviving tail
set message_count to the new live count
# same session id throughout. nothing is removed.Two flags, and the pair is doing more work than one flag could:
active |
compacted |
what it means | model sees it | search finds it |
|---|---|---|---|---|
| 1 | 0 | live | yes | yes |
| 0 | 1 | summarized away by compaction | no | yes |
| 0 | 0 | rewound. The user took it back | no | no |
That third row is the reason two flags beat one. “Gone from the model’s view” and “gone because the user retracted it” are different facts, and only one of them should still be findable. A single deleted bit cannot express that.
One detail makes the whole design affordable rather than merely nice. The trigger that keeps the search index in sync fires AFTER UPDATE OF content, tool_name, tool_calls, only those columns. Flipping active and compacted touches none of them, so archiving ten thousand messages fires no index work at all. It is one cheap UPDATE. Had that trigger been written on the whole row, every compaction would have rewritten the index for the entire conversation, and the honest version of this design would have been unaffordable.
The database stores a different string than the model saw
A detail that made me stop. A message row can hold two texts: the clean version a human reads, and a sidecar holding the exact bytes that were sent to the provider when those differed, because of an ephemeral memory injection, say.
On replay the sidecar is substituted back in verbatim, with no sanitizing and no stripping, and the comment is explicit that cleaning it would reintroduce the divergence it exists to remove. The cache contract again: the bytes must match what the provider saw last time, and the only way to guarantee that is to keep them.
The cost is a schema where “the message content” is genuinely ambiguous, and every export or display path has to decide which one it means.
Circuit-breaker state belongs on the row, not in the process
Remember the anti-thrash breaker from the compaction chapter, two strikes and it stops trying. Where do those strikes live?
On the session row, in columns, alongside the compaction failure cooldown and the fallback streak. There is one rehydration call that reloads all of them when a compressor binds to a session.
The bug that forced this is worth memorising, because most harnesses still have it. With the counter in memory, a restart reset it to zero, so a session whose prompt was permanently over the threshold would compact once per restart, forever, and each restart looked like a fresh start to the guard. Their own comment describes exactly that.
The rule generalises: any retry limiter, cooldown or thrash guard is broken until its counter outlives the process. And note the honest cost, a genuinely stuck session cannot be unstuck by restarting either. That is what a real breaker feels like.
The lesson hiding in micro-compaction
This one is my favourite, because it is a trap that only shows up much later.
Micro-compaction edits the message list in memory after a turn, folding an old exchange into a rolling summary. The persistence layer, meanwhile, is append-only: flushing a turn adds rows.
Put those two facts together and you get a bomb. The splice never happened as far as the database is concerned, so the original rows are still live, and on resume the harness loads the summary and the messages it summarized, which is worse than never having compacted at all. The code says so, and it fixes it by calling the archive path explicitly after every splice. When that call fails, it logs that resume will double-load until the next full compaction: graceful, and honestly lossy.
The general form: in-memory context editing and append-only persistence are not compatible by default. Every splice needs a matching archive write, or your resume path quietly reconstructs the conversation you were trying to shrink.
Two leases, and a hole
Multiple processes can hold the same conversation, the desktop app, a CLI resume, the gateway, a background delivery. So there are two leases: one per session for compaction, and one per conversation lineage for taking a turn. The lineage matters, because a conversation that has rotated into segments is still one thing that should not be run twice at once.
Both are re-validated inside the transaction that appends the message, which is the part to copy: a process that lost its lease during a long turn cannot land a row at the end of it.
Then there is the gap. The lease check runs only if a lease holder was supplied, and at least one write path supplies none: the delivery mirror, which appends a message into a session so the receiving side has context. So a cron job delivering an overnight brief can append into your transcript while your own turn is in flight. I have not traced every caller, so treat that as what the code shows rather than a bug report.
The rule is the sharp part: a fence enforced only when the caller opts in is not a fence.
The tool schema is built once, and that is a deliberate cost
Third fault: the tool schema is re-sent on every iteration. You would expect the harness to assemble that list per request, asking each tool whether it is currently available.
It does not. The list is computed once when the agent is constructed and never re-read from the registry. By now you can guess why, the tool block sits inside the cached prefix, so rebuilding it per request would invalidate the cache on every turn. Cheap freshness, expensive cache.
The cost of that choice is a whole bug family: a tool that becomes available later is invisible. An MCP server that finishes its handshake after startup, or reconnects, has tools nobody can call. Their answer is one named rebuild function with four callers, the terminal UI’s reload command, the gateway reload, a late-binding refresh thread, and a between-turns refresh, funnelled into one place, with a comment explaining that they had drifted apart before.
Availability checks, and the flake that amputated a toolbox
Whether a tool makes it into that snapshot is decided by a per-tool predicate: does Docker answer, is the browser driver installed, is there an API key. The result is cached for thirty seconds.
The cache is asymmetric, and it took a production incident to get there. A success is cached normally. A failure arriving within a minute of the last success is thrown away, the tool stays available, and the next call re-probes.
Why would you deliberately ignore a negative result from your own health check? Because the check is a subprocess with a timeout, and under load it flaps. A single Docker probe timing out used to strip the entire terminal and file toolset from whatever agent was being constructed at that instant, most visibly a freshly spawned child agent, which would then report that the file-reading tool does not exist. The tool had not gone anywhere. The probe had just been unlucky.
So the design accepts a worse failure mode to avoid a much worse one. For up to a minute after a genuine outage, the model is offered a tool that will fail when called, which costs a wasted turn, rather than being silently handed a smaller toolbox, which costs a confused agent that cannot explain why its own capabilities vanished.
Dispatch: not parallel, not serial, segmented
The model emits several tool calls at once. Do you run them in parallel or in order?
Neither, and the reframe is the lesson: this is a scheduling problem, so it gets a scheduler. The batch is walked in the order the model emitted it and cut into runs of calls that can safely overlap, separated by barriers that cannot.
for call in batch: # in the model's emission order
if call is interactive, unknown, or has unparseable args:
close the current run # a barrier: nothing overlaps it
elif call touches paths:
if its scope cannot be determined: close the run # a barrier too
if it overlaps a reservation and either side is a writer:
close the run
reserve its paths as reader or writer
add to the current run
elif call is on the read-only list:
add to the current run
run each segment in order; calls inside a run go concurrently
results are reassembled in emission order, whatever finished firstThe path reservation is what makes this safe, and the rule is one sentence from the source: concurrent reads of the same subtree commute, so reader against reader never conflicts, while a writer conflicts with any overlap. That is what stops the classic race where a model batches a file read alongside the patch it depends on, and the read lands first.
Two details that show the thinking. A search tool reserves its search root as a reader, so a search batched after a write into that tree is ordered behind it. And a “parallel” run holding only one call is demoted to sequential, because there is no concurrency to win and the sequential path has richer handling.
The price is visible: an allowlist of about a dozen read-only tool names maintained by hand, real path canonicalization on every scoped call, and enough concurrency machinery that they needed a gate to keep approval prompts appearing in the model’s call order. That gate has a bounded wait, and when it expires it proceeds out of order on purpose. The comment is refreshingly plain about the trade: interleaved approval prompts are strictly better than permanent starvation.
Large output gets relocated, not truncated
A single tool result can be enormous, a terminal dump, a large file, a web extraction. Three layers handle it: each tool caps its own output, then any result over about a hundred thousand characters is spilled, then if many medium results still add up past the per-turn budget, the largest are spilled until the total fits.
Spilled means written to a file in the sandbox and replaced in the conversation with a preview of about fifteen hundred characters, the character count, the path, and an instruction to page it back in with offsets. Nothing is lost, and no model call is spent condensing it. Both budgets scale down with the model’s context window, and never up.
There is a wonderful pin in that config. The file-reading tool registers its own hundred-thousand-character cap, and the per-result layer overrides that one tool to infinity. It has to: the recovery instruction says read the file, so a capped reader would spill the recovery read to a new file and tell the model to read that one, forever. The moment your recovery path names a tool, that tool’s own limits become a recursion hazard.
Note that the pin only binds that layer; the per-turn sweep can still spill a large read under a synthetic name.
One asymmetry to keep in mind if you ever debug this from a transcript: the display surfaces get the full tool output, because that copy is taken before the spill, while the model sees the preview. The plain terminal line truncates too, so a report that the agent “ignored” something it was obviously shown may be a report that the agent never saw it.
The floor is one they chose
All of that machinery is measured against a threshold, and the threshold has to clear a floor no amount of compaction can move: the system prompt plus the tool schemas, sent in full on every request.
Their reasoning for its size is good and written down. The core is described as a narrow waist with capability at the edges, the second of the two review lenses, and the stated reason is the right one: every tool they add is sent on every call, so the bar for a new core tool is high. A ladder pushes new capability toward a CLI command plus a skill, then service-gated tools, then plugins, with a core tool as the last resort. Most projects have nothing like it, and the always-available surface is the product: one agent that does the thing wherever you are talking to it, which you cannot ship with eight tools.
And yet the list runs to fifty-nine, and their own comment budgets the resulting floor at twenty to thirty thousand tokens with fifty-plus tools. Under this harness’s own economics that floor is cheap in money, since it sits ahead of every rewrite point and is re-read at a tenth of the price. What it costs is window, and therefore latency: it is why the compaction trigger has to be raised from half to three-quarters on models under a 512K window, and their comment says what happens otherwise, which is that the floor eats most of the reclaimed headroom, compaction re-fires every turn or two, and the session spends its wall clock summarizing.
So what does this harness optimize for? Not input-token count, which it will let grow rather than rewrite history. It optimizes for the turn economics of a long-lived conversation, and specifically for the cached prefix. Not cost per successful task either, though the calibration loop and the breaker come closer than most, since both exist to stop it spending money on work that produces nothing. The gap is precise: these mechanisms catch wasted harness work rather than wasted task work.
Nothing here notices that the agent got the answer wrong and is about to try again.
I have made a version of this argument about 23 models that were an architecture problem in a modeling costume; here the count is the price of the product promise, and none of the cache discipline reduces it.
One more comment deserves the last word here. When a model emits a tool call with an empty name, which weak models do after reading tool-call syntax in a file and echoing it, the harness deliberately does not reply with the list of valid tools: sending the catalog feeds the imitation loop more names to mimic and inflates the context several times over across retries. A genuine typo still gets the catalog.
Your error messages are part of the model’s input distribution, and a helpful one can be the thing sustaining the failure.
The fourth fault: a tool call will delete your files if asked
Every dangerous command goes through one function, and the order of its checks is the security design.
Three things fire before any bypass is consulted: a hardline floor of unrecoverable commands, a guard against the model piping guessed passwords into sudo, and the user’s own deny list. Only then does the code look at whether you turned approvals off. The comment is the clearest statement of intent in the file: opting into the trust-everything mode is trusting the agent with your files and services, not trusting it to wipe the disk or power off the box.
Worth knowing what this costs before we go further. A human gate is a blocking wait, with a five-minute default timeout, and the harness had to build an accounting layer so that time parked on a person does not count against the deadline of a concurrent tool batch. Latency is the price of controllability, and here it is paid in whole minutes.
The sudo guard deserves its own note, because it is a different kind of thinking. Piping a password to sudo -S when no password is configured is not a destructive command; it is the model brute-forcing, using the “Sorry, try again” output as an oracle and its iteration budget as the loop. The guard is not detecting damage. It is detecting that the agent is in a position to iterate against a feedback signal.
The one-line attack, and the fix that costs four comments
You can see the attack yourself if I show you the wrong version first. Imagine the bypass check read the environment on every call:
# the tempting version
on every check: if env["BYPASS_APPROVALS"]: allow # DON'T
# what ships
when code loads: BYPASS = truthy(env["BYPASS_APPROVALS"]) # read once, frozen
on every check: if BYPASS: allowThe agent runs skills, plugins and generated code inside its own process. So with the first version, one line of injected Python setting that variable silently disables every approval check for the rest of the process. No prompt, no log, nothing to notice.
That single change inverts the timing requirement. To enable the bypass you must set the variable before the approval module is imported, which is during process startup, before any model output exists. Model-generated code, by construction, runs after. The environment variable stops being a runtime switch and becomes a launch-time argument the model cannot reach.
The cost is paid in plumbing, and honestly: because --yolo must now be materialized before tool discovery triggers that import, five launcher paths set it, most of them carrying a comment about the ordering, and one issue exists because a launcher was missed. There is a per-session toggle for runtime use that routes through an audited path instead.
The general form is worth writing down: security-relevant configuration read on the hot path is attacker-writable configuration, whenever the attacker can run code in your process. Which, in an agent harness, they can.
The gate that trust cannot switch off
Almost every guard here protects against damage. One protects against something else, and it is the idea I would take furthest.
Writes to files that steer future behaviour, the agent instruction files a project loads on startup, always require human approval. Every time, one operation at a time, with no persistent grant, and even when you have turned approvals off entirely. It deliberately refuses to route through the shared approval gate, because that gate honours the bypass and allowlists, and the whole point is that this one does not.
The reasoning: an injected instruction that edits those files outlives the turn and poisons every later session that reads them. Damage is bounded by a session. A rewritten instruction file is not.
That distinction is why the same file also hard-blocks writes to the harness’s own config, the file where “approvals: off” lives. Since the approval mode is re-read live on every check, a single successful write there would collapse the entire ladder on the next command.
What is not here: a sandbox
Worth being plain, because it shapes everything above. Hermes installs no operating-system sandbox on the default path. No seatbelt profile, no Landlock, no namespaces, no seccomp; I looked for all of them, and the only hits are build and test tooling. The default terminal backend executes on your host.
Container backends, remote backends and one opt-in runtime that hands the whole turn to another vendor’s sandboxed subprocess all exist as options, so the isolation is borrowed rather than absent. And the trade is explicit in a way I have not seen elsewhere: when the backend is isolated, the approval layer is skipped entirely, including the hardline floor. rm -rf / inside a container with no host mounts is approved silently, and correctly. There is nothing there to lose.
Docker with host paths mounted goes back through approvals, because now the command reaches your files.
So isolation and approval are substitutes here rather than layers. Run isolated and the sandbox is the boundary; run on the host, which is the default, and the boundary is a pattern list plus your attention. There is no defence in depth on that path: a detection miss is a total miss.
One more thing to carry away, from a pair of security advisories rather than a design doc. Session identity for approvals used to live in an environment variable, which is process-global, so with several sessions sharing a thread pool, one session’s cleanup could clobber another’s mid-run, dropping it onto the branch where no human is present and dangerous commands auto-approve. The fix was context-local identity. But context-local state does not cross a bare thread boundary either, so every place that fans out work had to be taught to carry it, which was the second advisory. The lesson is not “use contextvars”. It is: when a security decision depends on ambient state, enumerate every boundary that state must cross, and make missing state mean deny.
Delegation: a child that starts empty
Delegation is not on that list of four. It answers a fifth problem the loop cannot solve on its own: a task too large for one context. The answer is a child agent, and what it does not inherit is the design.
A child gets a fresh conversation with none of the parent’s history, its own task id, the parent’s toolset minus a blocklist, and a focused prompt built from the delegated goal. The parent sees only the delegation call and the summary that comes back, never the child’s intermediate tool calls or reasoning. That is the point: delegation is context isolation, and the parent pays a summary rather than a transcript.
Five tools are stripped from every leaf child, which under the shipped spawn depth of one means every child, and each closes a distinct axis: no delegating (no fan-out amplification), no asking the user (nothing competes for one stdin), no memory writes (no racing on shared long-term state), no messaging (no side effects the parent cannot see), no scheduling (no work outliving the session in the parent’s name). A child can never gain a tool the parent lacks. Raise the spawn depth and an orchestrator child gets delegation back, which is the one axis on that list you can re-open.
The parent’s summary budget is the detail I would copy: half of the parent’s remaining context headroom, divided across the batch, floored so a single summary is never truncated to noise, and spilled to a file if it overflows. Ten children cannot collectively blow the window that the parent still has to think in.
Two honest limits. The parent is told, in the tool description, that child summaries are self-reports rather than verified facts, if a child claims it uploaded a file, get a handle and check it yourself. And background delegation is durable in its reporting, not its execution: a completed result survives a crash and is delivered later, but a child in flight when the process exits is simply gone, and the parent learns only that the outcome is unknown.
Providers: the part of the abstraction that held, and the part that didn’t
Every request we have followed so far ends at a provider, and support for them is two layers. Thirty-odd providers are declared as data, a dataclass naming auth, endpoints, headers, catalog quirks, and four transports own the wire shapes: an OpenAI-style chat completion, Anthropic’s messages, a responses-style envelope, and Bedrock. Adding a provider is somewhere between a dozen and forty lines of configuration. That layer worked.
The layer that did not hold is failure. Provider names are checked over two hundred times inside the agent core. The single object that tracks turn recovery carries a one-shot flag per provider for credential refresh, plus more named after individual models’ format bugs. Two of the six wire modes have no transport at all and are branched on directly in the loop.
You could read that as sloppiness. I think it is the more interesting thing: a provider’s identity is not just how you call it, it is how it breaks. Which of the three meanings a 429 has this time, your key is throttled, the whole endpoint is overloaded, or an aggregator’s upstream is throttling, determines whether the right move is rotate the credential, back off on the same key, or switch models. That is not declarable.
It is learned from a response body at runtime, and there is no slot in a config dataclass for it.
So instead of hiding failures, they enumerated them. A classifier maps errors into a taxonomy that returns recovery hints rather than a label: is it retryable, should we compress, should we rotate the credential, should we fall back. The loop reads booleans and never re-classifies. Deterministic failures, a content-policy refusal, a bad certificate, a malformed model id, are explicitly marked non-retryable so they fail in one attempt instead of burning three.
The struct that makes cost accounting possible
One small decision here pays for itself many times over. Usage is normalized into four disjoint priced buckets, fresh input, output, cache reads and cache writes, and the familiar prompt-token total is computed from them rather than stored. Reasoning tokens ride along as a fifth count, but they are a breakdown of output rather than a peer of it, and nothing prices them.
Providers disagree about this. Anthropic reports input exclusive of cache; the OpenAI and responses shapes report an inclusive total with a breakdown alongside. So the normalizer subtracts on the way in, and everything downstream sees parts that do not overlap.
The payoff is that pricing multiplies each bucket by its own rate and double-charging a cached token is structurally impossible, not merely avoided. And the harness will refuse to produce a number it cannot defend: if usage has tokens in a bucket whose rate is unknown, the whole result comes back as unknown rather than as a partial sum, and every cost figure carries where its rate came from and when it was fetched.
There is a lesson sitting next to it in the same repo. The transport layer defines its own usage struct, one that stores the provider’s inclusive total plus a cached count, and nothing in production reads it. It cannot price a request correctly, so the real work happens elsewhere. That is what an abstraction drawn before its hardest requirement was understood looks like, and the tell is that it compiles, passes tests, and is dead.
For completeness, since you may be waiting for it: there is no spend cap. There is a warning when you select an expensive model, and no mechanism that halts a run at a dollar figure. Unattended cron work has no financial stop.
Telemetry that cannot cost you anything
The gateway and cron emit events through one queue with a contract stated as an invariant: the emit call must return in microseconds, must never block on disk or network, and must never raise into the caller. When the buffer is full it drops the oldest event and counts the drop, and a daemon thread fans out to subscribers that are individually isolated, so a slow collector cannot reach back into a turn.
The privacy half is the part I would copy. The built-in events are not content-free by policy. They are content-free because the event types have no field a message could go in. Redaction runs anyway and fails closed: if the redactor cannot run, the string is replaced rather than emitted. Anything carrying conversation content lives in a separate opt-in plane.
What it does not measure
The verification story is where I think this repo falls short of its own standards, and it is worth ending the tour on an honest note.
There is an enormous unit suite, thousands of test files, and it is genuinely good at pinning mechanism: does this classifier return overloaded for this body, does that trigger fire at this threshold. There is also one agent-level eval, for the file-reading tool, and it is better than most published benchmarks: it runs the real agent against deliberately hostile fixtures, grades against planted ground truth, and ships rules of engagement that say single-run differences under three percent are noise and every measurement needs three repetitions.
What does not exist is a harness-wide task suite. Trajectories are exported, but in a training format. This is a lab, and those traces feed model work rather than replay debugging.
So the parts of this system I have spent nine chapters admiring are, precisely, the parts that unit tests and code review can see. A refactor that keeps every test green and makes the agent worse at multi-step work would ship undetected. That is not a small gap in a codebase this careful, and it is the fair counterweight to all of it.
What the coverage adds up to
That is the tour. The synthesis lives next door: the three mechanisms I have not seen in another harness, the tradeoffs gathered in one table, the one idea I would take and the one trap that will bite you are all in the shorter companion piece, which is the version to read if you came here for the argument rather than the map.
What the coverage itself teaches is something no single mechanism does. Go looking through this repo and you keep finding a particular genre of comment: one that explains why the obvious fix is wrong, and names the bug it caused. The non-atomic cache write that silently erased every learned context length. The reasoning-strip applied to the wrong list, which corrupted stored conversations. The probe ladder that turned a user’s million-token window into a quarter of it. Those comments record the shape of a failure, which is the thing a passing test cannot tell you and the thing you cannot recover by reading the code as it stands.
The question I would put to the next harness comes from the shape of this one. Its durable record is already append-only in the sense that matters: a compaction archives rows rather than deleting them, and they stay searchable under the same session id. What gets rewritten is the outgoing request, and that is what every gate, cooldown and durable counter in this post exists to protect. So why is the request assembled by editing history at all, rather than projected from it, so that what the model sees can change without changing a byte of what the provider has already seen?
I’m Kaushal Prajapati, a Staff AI/ML Engineer. More posts, or say hi on LinkedIn.